🤖 SEO Tool
Robots.txt
Generator
Set crawler rules visually and download a ready-to-use robots.txt file instantly.
Crawler Rules
Rule #1
robots.txt Preview
User-agent: * Disallow: /admin/
Upload this file to your website root: https://yourdomain.com/robots.txt
Frequently Asked Questions
What is a robots.txt file?▾
robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages or sections they should or should not crawl and index.
What is the difference between Allow and Disallow?▾
Disallow blocks crawlers from accessing a path. Allow is used to override a broader Disallow rule for a more specific path. For example, you can disallow /private/ but allow /private/public/.
What is Crawl-delay?▾
Crawl-delay tells crawlers how many seconds to wait between requests. It helps reduce server load. Note: Googlebot ignores Crawl-delay, but Bingbot and Yandex respect it.
Where should I upload my robots.txt?▾
It must be placed at the root of your domain: https://yourdomain.com/robots.txt. It will not work in subdirectories.