Robots.txt Generator
Build a clean robots.txt file in seconds. Tell search engines which parts of your site they can crawl and which to skip.
Choose allow-all or restricted
Most sites want to allow everything except admin areas.
Optionally list paths to disallow
One per line. Hide admin, search, draft pages.
Add your sitemap URL
Critical for indexing.
Build robots.txt
What robots.txt does (and what it does not)
Robots.txt is a plain-text file at the root of your website that tells search engine crawlers which areas they should not crawl. It is the politest fence on the internet — well-behaved bots respect it, malicious ones ignore it. Never use robots.txt to hide sensitive data; use authentication for that.
Common patterns
- Allow everything:
Disallow:with nothing after = crawl everything. - Block admin areas:
Disallow: /admin/stops crawlers from indexing admin pages. - Block search results:
Disallow: /searchstops Google from indexing internal search pages. - Always link your sitemap: Helps search engines find all your pages.
Disallow: /
Blocks the entire site. Useful only during dev. Disaster on production.Disallow: /secret-password.txt
Robots.txt is public — you just told everyone where the secret is.User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
Allow all, expose sitemap. Standard for most sites.User-agent: * Disallow: /admin/ Disallow: /search
Block admin and internal search.Frequently asked questions
Where do I put robots.txt?
In the root of your website, so it is reachable at https://yoursite.com/robots.txt. Subfolders do not work.
Does robots.txt prevent indexing?
No. It prevents crawling. A page blocked in robots.txt can still be indexed if Google finds it linked elsewhere. To prevent indexing, use meta robots noindex.
Should I add a Crawl-delay?
Usually no. Google ignores Crawl-delay. Only some crawlers honor it. Add it only if your server is being hammered.
What is the difference between Allow and Disallow?
Disallow blocks a path. Allow explicitly permits a path. Allow is useful to carve out exceptions inside a blocked directory.
Do I need to list every crawler separately?
No. User-agent: * matches all crawlers. List specific user-agents only when you want different rules for different bots.