Free tool
AI crawler robots.txt builder
Some AI bots collect pages for training, some index them for AI search, and some fetch a page because a person asked. Pick which to keep out and copy the rules into your robots.txt. Each bot’s description comes from its operator’s own documentation, checked November 14, 2026.
Your robots.txt
# robots.txt built with https://bluenebula.dev/tools/ai-crawler-robots-txt # AI crawlers kept out of the whole site User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Amazonbot Disallow: / # Everyone else, including the AI crawlers not listed above User-agent: * Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
robots.txt only asks. Crawlers that follow the standard respect it, and it can't stop the ones that don't.
Training, search and fetches
The big AI companies now run separate bots for separate jobs. OpenAI, for example, has GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages someone asks about. Anthropic splits Claude the same way. That means you can stay out of training data and still appear in AI search answers, which is the setting the builder starts with.
Two of the tokens aren’t crawlers at all. Google-Extended and Applebot-Extended only tell Google and Apple whether pages their regular crawlers already fetch may train their models, so blocking them doesn’t change how you appear in Google or Apple search.
For the other direction, telling AI assistants what your site is about, see the llms.txt generator.
Questions
Does blocking GPTBot keep my site out of ChatGPT?
Not out of ChatGPT search. OpenAI uses GPTBot for training. Its search crawler is OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers. Block GPTBot and allow OAI-SearchBot if you want to stay out of training but still be found.
Does blocking Google-Extended affect Google Search?
Google says it doesn't: Google-Extended only controls whether pages Googlebot crawls may be used for Gemini, and it isn't a ranking signal. It doesn't crawl anything itself.
Will robots.txt stop every AI bot?
No. RFC 9309, the robots.txt standard, says its rules are not a form of access authorization. OpenAI, Perplexity and Meta also say the fetchers that open a page because a person asked may not follow robots.txt. If you need a hard block, use your host's firewall.
Why does the builder only name the crawlers I block?
Under RFC 9309 a crawler uses the group with its own name and ignores the * group. Naming only the blocked crawlers means every allowed one keeps following your normal rules, including any paths you keep everyone out of.
Why isn't Bytespider on the list?
ByteDance doesn't publish documentation for it, so there's nothing reliable to say about what it does or whether it follows robots.txt. Every crawler on the list links to its operator's own page.
Sources
- RFC 9309: Robots Exclusion Protocol (IETF, 2022)
- OpenAI: crawler documentation
- Anthropic: crawler documentation
- Google: crawler documentation
- Apple: crawler documentation
- Meta: crawler documentation
- Common Crawl: crawler documentation
- Amazon: crawler documentation
- Perplexity: crawler documentation
- DuckDuckGo: crawler documentation
- Mistral: crawler documentation