Robots.txt Validator
Paste your robots.txt and get instant analysis. Catches syntax errors, missing User-agent declarations, ignored Crawl-delay, missing sitemap reference, and more.
Paste robots.txt
The full content of your robots.txt file.
Click validate
We check syntax and best practices.
Fix issues found
Use our Robots.txt Generator if needed.
Validate robots.txt
What this validator checks
- Syntax errors: Missing colons, malformed directives, invalid line structure.
- Missing User-agent: Robots.txt without User-agent is non-functional.
- Misplaced directives: Allow/Disallow without a preceding User-agent.
- Sitemap presence: Best practice is to include a Sitemap: line.
- Crawl-delay warnings: Google ignores Crawl-delay; we warn you.
- Unknown directives: Typos or non-standard rules.
- Site-wide Disallow: Disallow: / blocks the entire site — verify intent.
Frequently asked questions
Where do I get my robots.txt?
Visit yoursite.com/robots.txt in any browser. Copy the entire content into this validator.
What is the difference between an error and a warning?
Errors break functionality (robots.txt does not work as intended). Warnings are best-practice issues that may not break anything but are worth fixing.
Why does it warn about Crawl-delay?
Google explicitly ignores Crawl-delay. Some crawlers (Bing, Yandex) honor it, but it is not standard. Better to manage crawl budget through Search Console.
Does this validator follow the latest standards?
Yes. We follow Google's 2024 robots.txt specification (now an official IETF standard, RFC 9309).
Can I have multiple User-agent blocks?
Yes. Common pattern: rules for User-agent: * (all bots), then specific overrides for User-agent: Googlebot, etc.