Robots.txt generator
Free robots.txt generator. Build allow/disallow rules, crawl-delay and a sitemap line, block AI crawlers in one click, then copy or download — all in your browser.
100% private — files never leave your devicePlace this file at the root of your domain: /robots.txt
How to use
- Pick a preset or start custom. Choose allow-all, block-all or WordPress defaults as a starting point, or build your rules from scratch.
- Add your rules. Set the user-agent, then list the paths to Disallow and Allow (one per line), plus an optional crawl-delay.
- Add your sitemap and AI blocking. Paste your sitemap URL, and tick the box to also block AI crawlers like GPTBot and CCBot if you want.
- Copy or download. Copy the generated robots.txt or download the file, then upload it to the root of your domain.
Features
- Live, valid output. The robots.txt updates as you type, with correct User-agent, Disallow, Allow, Crawl-delay and Sitemap syntax.
- One-click AI-crawler blocking. Add rules to block GPTBot, CCBot, Google-Extended, ClaudeBot, PerplexityBot and more in a single click.
- Handy presets. Allow all, block all, or WordPress defaults give you a correct starting point instantly.
- Sitemap + crawl-delay. Include your sitemap URL and an optional crawl-delay for bots that honour it.
- 100% private. Everything is generated locally in your browser — nothing you enter is uploaded or stored.
When to use it
- Creating a robots.txt for a brand-new website
- Blocking AI crawlers from training on your content
- Hiding admin, cart or search pages from search engines
- Adding your sitemap so crawlers find every page
- Fixing an incorrect or empty robots.txt file
Is it private?
Your robots.txt is generated entirely in your browser — nothing you enter is uploaded or stored.
Frequently asked questions
Is this robots.txt generator free?
Yes — completely free, unlimited and private. The file is generated in your browser with no sign-up and nothing uploaded.
How do I block AI crawlers?
Tick the 'block AI crawlers' option and the tool adds Disallow rules for GPTBot, CCBot, Google-Extended, ClaudeBot and other AI bots. Note that compliance is voluntary — well-behaved bots honour it.
Where does robots.txt go?
At the root of your domain, reachable at https://yourdomain.com/robots.txt. Search engines look for it there automatically.
Does Disallow hide a page from Google?
It asks crawlers not to crawl a path, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex meta tag instead.
Should I add my sitemap here?
Yes — a Sitemap line helps crawlers discover all your URLs. You can also submit the sitemap directly in Google Search Console and Bing Webmaster Tools.
Is my input uploaded anywhere?
No. The file is built entirely in your browser, so nothing you type leaves your device.
Last updated