Skip to content
Blue Nebula

Free tool

AI crawler robots.txt builder

Some AI bots collect pages for training, some index them for AI search, and some fetch a page because a person asked. Pick which to keep out and copy the rules into your robots.txt. Each bot’s description comes from its operator’s own documentation, checked November 14, 2026.

Start from:
Training

Collect pages to train AI models. Blocking these doesn't remove you from AI search.

  • Crawls content that may be used to train OpenAI's generative AI models.

    Says it follows robots.txt · Docs

  • Collects pages for Anthropic's model training; blocking it excludes a site's future pages.

    Says it follows robots.txt · Docs

  • A control token, not a separate crawler: it decides whether pages Googlebot crawls may train Gemini. It doesn't affect Google Search.

    Says it follows robots.txt · Docs

  • Doesn't crawl: it decides whether pages Applebot crawls may train Apple's foundation models. Blocked pages can still appear in search.

    Says it follows robots.txt · Docs

  • Crawls the web for uses such as training AI models or indexing content for Meta's products.

    Doesn't say · Docs

  • Builds Common Crawl's open archive of the web, a dataset many others download.

    Says it follows robots.txt · Docs

  • Improves Amazon's products and services, and may be used to train Amazon's AI models.

    Says it follows robots.txt · Docs

AI search

Index pages so AI search can show and link them. Blocking these can keep you out of those answers.

  • Surfaces websites in ChatGPT search. Sites that block it aren't shown in ChatGPT search answers.

    Says it follows robots.txt · Docs

  • Reads pages to improve the relevance and accuracy of Claude's search results.

    Says it follows robots.txt · Docs

  • Surfaces and links websites in Perplexity's search results. Perplexity says it isn't used to train models.

    Says it follows robots.txt · Docs

  • Crawls pages in real time for DuckDuckGo's AI-assisted answers. Changes take effect after 72 hours.

    Says it follows robots.txt · Docs

Fetches for a person

Open a page because someone asked an assistant to. Several operators say these may ignore robots.txt.

  • Visits a page when someone asks ChatGPT or a custom GPT a question. OpenAI says robots.txt rules may not apply.

    May ignore robots.txt · Docs

  • Fetches pages for people using Claude. Blocking it can reduce your visibility in searches people run through Claude.

    Says it follows robots.txt · Docs

  • Fetches a page when a Perplexity user asks for it. Perplexity says it generally ignores robots.txt.

    May ignore robots.txt · Docs

  • Fetches individual links at a person's request. Meta says it may bypass robots.txt.

    May ignore robots.txt · Docs

  • Visits a page when someone asks Mistral's assistant a question.

    Doesn't say · Docs

Your robots.txt

# robots.txt built with https://bluenebula.dev/tools/ai-crawler-robots-txt

# AI crawlers kept out of the whole site
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Amazonbot
Disallow: /

# Everyone else, including the AI crawlers not listed above
User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

robots.txt only asks. Crawlers that follow the standard respect it, and it can't stop the ones that don't.

Training, search and fetches

The big AI companies now run separate bots for separate jobs. OpenAI, for example, has GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages someone asks about. Anthropic splits Claude the same way. That means you can stay out of training data and still appear in AI search answers, which is the setting the builder starts with.

Two of the tokens aren’t crawlers at all. Google-Extended and Applebot-Extended only tell Google and Apple whether pages their regular crawlers already fetch may train their models, so blocking them doesn’t change how you appear in Google or Apple search.

For the other direction, telling AI assistants what your site is about, see the llms.txt generator.

Questions

Does blocking GPTBot keep my site out of ChatGPT?

Not out of ChatGPT search. OpenAI uses GPTBot for training. Its search crawler is OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers. Block GPTBot and allow OAI-SearchBot if you want to stay out of training but still be found.

Does blocking Google-Extended affect Google Search?

Google says it doesn't: Google-Extended only controls whether pages Googlebot crawls may be used for Gemini, and it isn't a ranking signal. It doesn't crawl anything itself.

Will robots.txt stop every AI bot?

No. RFC 9309, the robots.txt standard, says its rules are not a form of access authorization. OpenAI, Perplexity and Meta also say the fetchers that open a page because a person asked may not follow robots.txt. If you need a hard block, use your host's firewall.

Why does the builder only name the crawlers I block?

Under RFC 9309 a crawler uses the group with its own name and ignores the * group. Naming only the blocked crawlers means every allowed one keeps following your normal rules, including any paths you keep everyone out of.

Why isn't Bytespider on the list?

ByteDance doesn't publish documentation for it, so there's nothing reliable to say about what it does or whether it follows robots.txt. Every crawler on the list links to its operator's own page.

Sources

  1. RFC 9309: Robots Exclusion Protocol (IETF, 2022)
  2. OpenAI: crawler documentation
  3. Anthropic: crawler documentation
  4. Google: crawler documentation
  5. Apple: crawler documentation
  6. Meta: crawler documentation
  7. Common Crawl: crawler documentation
  8. Amazon: crawler documentation
  9. Perplexity: crawler documentation
  10. DuckDuckGo: crawler documentation
  11. Mistral: crawler documentation

Need help with your site's crawl rules?

Tell me what you’re building.

mail@bluenebula.dev
Start a project