Robots.txt Generator & Checker
Draft crawler rules for your site and check for a full-site block before publishing.
Your file contains a user-agent and a valid HTTPS sitemap URL.
This checker validates common syntax and risky rules. Always test important URLs in Google Search Console before publishing.
One slash can change the scope of a rule
User-agent: *
Disallow: /drafts/Asks matching crawlers not to fetch the drafts section.
User-agent: *
Disallow: /Asks matching crawlers not to fetch any path on the site.
Enter one path per line and read the complete generated file before publishing it at the site root. This builder adds a leading slash when needed and warns about a whole-site block, a missing user-agent or a sitemap address without HTTPS.
Robots rules are not access control — a disallowed page can still be accessed directly and may still appear in search results. Protect private content with authentication, not a robots rule. Google explains these limits.
Does “Basic validation passed” mean every path is allowed correctly?
No. The checker reviews the fields you entered; it does not fetch your live robots.txt or simulate each crawler against a URL. Check the actual file and your important pages after deployment.