In short
- The list names 26 tokens from 10 operators. Every row links to the operator's documentation.
- The major providers separate training, search and fetching on user request. You can block training and still appear as a source in AI search.
- Bots that fetch a page at a user's request often follow robots.txt only to a limited extent.
- List as of 15 Oct 2026.
Four types of AI bots
What an operator uses the content for determines what a block in robots.txt achieves. The list is therefore sorted by purpose.
- Training. The bot collects content for future models.
- Search and index. The bot builds an index from which AI search tools choose their sources.
- Fetching on user request. The bot fetches a page because someone is asking about it right now.
- Control tokens and special cases. Tokens without their own crawler and bots with a narrowly defined task.
What these types mean for your robots.txt, and what a rule that blocks training but allows AI search looks like, is explained in the guide AI crawlers.
Training
| robots.txt token | Operator | What the bot does | robots.txt | Source |
|---|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's foundation models | According to OpenAI, Disallow means content should not be used for training | Docs undated, retrieved 15 Oct 2026 |
| ClaudeBot | Anthropic | Collects content that may be used to train Claude models | yes, including Crawl-delay | Docs as of 7 Apr 2026 |
| meta-externalagent | Meta | Crawls to train foundation models and to index content for Meta products | Meta shows a robots.txt rule for this token | Docs undated, retrieved 15 Oct 2026 |
| MistralAI-Training | Mistral AI | Collects content for the training data of Mistral models | yes, can be blocked with Disallow | Docs undated, retrieved 15 Oct 2026 |
| Amazonbot | Amazon | Improves Amazon products; the data may be used to train Amazon models | yes, without Crawl-delay | Docs undated, retrieved 15 Oct 2026 |
| CCBot | Common Crawl | Builds an open web archive that is often used as a basis for training | yes | Docs undated, retrieved 15 Oct 2026 |
Search and index
Googlebot and Applebot serve several purposes. If you block them, you also disappear from classic search.
| robots.txt token | Operator | What the bot does | robots.txt | Source |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Shows websites as sources in ChatGPT search | yes, changes take effect after around 24 hours | Docs undated, retrieved 15 Oct 2026 |
| Claude-SearchBot | Anthropic | Indexes content to improve search results in Claude | yes | Docs as of 7 Apr 2026 |
| PerplexityBot | Perplexity | Shows and links websites in Perplexity search results; according to Perplexity, no training | yes | Docs undated, retrieved 15 Oct 2026 |
| MistralAI-Index | Mistral AI | Indexes content for Mistral search; according to Mistral, no training | yes | Docs undated, retrieved 15 Oct 2026 |
| Amzn-SearchBot | Amazon | Indexes for search features in Amazon products such as Alexa; according to Amazon, no training | yes; without its own group, the rules for other search bots apply | Docs undated, retrieved 15 Oct 2026 |
| meta-webindexer | Meta | Indexes for search in Meta AI so that Meta AI can cite and link content | Meta names robots.txt as the control | Docs undated, retrieved 15 Oct 2026 |
| DuckAssistBot | DuckDuckGo | Fetches pages for DuckDuckGo's AI-assisted answers; according to DuckDuckGo, no training | yes, changes take effect after 72 hours | Docs undated, retrieved 15 Oct 2026 |
| Googlebot | Crawls for Google Search, which includes AI Overviews and AI Mode | yes | Docs as of 10 Dec 2025 | |
| Applebot | Apple | Crawls for Spotlight, Siri and Safari; the data may also be used to train Apple models | yes, without Crawl-delay; without its own group, the rules for Googlebot apply | Docs as of 4 Sept 2026 |
Fetching on user request
These bots do not crawl on their own. They fetch a page when a person asks about it in the assistant or shares a link. Several operators state that robots.txt does not always apply in this case.
| robots.txt token | Operator | What the bot does | robots.txt | Source |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | Fetches a page when someone asks about it in ChatGPT or a custom GPT | according to OpenAI, the rules may not apply | Docs undated, retrieved 15 Oct 2026 |
| Claude-User | Anthropic | Fetches a page when someone asks Claude a question | yes, a block prevents the fetch | Docs as of 7 Apr 2026 |
| Perplexity-User | Perplexity | Fetches a page when someone asks a question in Perplexity | generally no | Docs undated, retrieved 15 Oct 2026 |
| MistralAI-User | Mistral AI | Fetches a page when someone asks a question in Mistral Vibe | yes, the token controls which pages are fetched | Docs undated, retrieved 15 Oct 2026 |
| Amzn-User | Amazon | Fetches up-to-date information, for example for Alexa answers | not necessarily all rules | Docs undated, retrieved 15 Oct 2026 |
| meta-externalfetcher | Meta | Fetches individual links at a user's request, for example for AI agents | can bypass the rules | Docs undated, retrieved 15 Oct 2026 |
| Google-Agent | Agents on Google infrastructure that operate websites on behalf of a user | Google states generally that user-triggered fetches usually ignore the rules | Docs as of 19 Aug 2026 |
Control tokens and special cases
| robots.txt token | Operator | What the bot does | robots.txt | Source |
|---|---|---|---|---|
| Google-Extended | Control token without its own crawler. Controls whether content is used to train future Gemini models and for grounding in Gemini and Vertex AI. No effect on Google Search. | robots.txt only | Docs as of 15 Oct 2026 | |
| Applebot-Extended | Apple | Control token without its own crawler. Controls whether Applebot data is used to train Apple models. | robots.txt only | Docs as of 4 Sept 2026 |
| Google-CloudVertexBot | Crawls on behalf of website operators who build agents in Vertex AI. No effect on Google Search. | yes | Docs as of 15 Oct 2026 | |
| OAI-AdsBot | OpenAI | Checks landing pages submitted as ads in ChatGPT; according to OpenAI, no training | not specified | Docs undated, retrieved 15 Oct 2026 |
Verifying authenticity
A user agent is easy to fake. Most operators therefore publish the IP ranges of their bots. Compare the address from your log with these lists before you attribute a request to an operator.
| Operator | IP lists and verification method |
|---|---|
| OpenAI | searchbot.json, gptbot.json, chatgpt-user.json, adsbot.json |
| Anthropic | bots.json |
| Perplexity | perplexitybot.json, perplexity-user.json |
| common-crawlers.json, user-triggered-agents.json | |
| Apple | applebot.json, reverse DNS to *.applebot.apple.com |
| Amazon | Amazonbot, Amzn-SearchBot, Amzn-User |
| Common Crawl | ccbot.json, reverse DNS to *.crawl.commoncrawl.org |
| DuckDuckGo | duckassistbot.json |
| Mistral AI | linked on the documentation page |
Anthropic points out that blocking by IP address can prevent the opt-out, because the bot can then no longer read your robots.txt.
Not on the list
Older lists name further identifiers, for example anthropic-ai, Claude-Web, Bytespider, cohere-ai or YouBot. On 15 Oct 2026 we found no current operator documentation for these identifiers. They are therefore not in the table.
Is a bot missing or has a detail changed? Write to us at info@searchviu.com.
Sources
- OpenAI: Overview of OpenAI Crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? (7 Apr 2026)
- Perplexity: Perplexity Crawlers
- Google: Google common crawlers (15 Oct 2026)
- Google: Google user-triggered fetchers (19 Aug 2026)
- Google: AI features and your website (10 Dec 2025)
- Apple: About Applebot (4 Sept 2026)
- Meta: Meta Web Crawlers
- Amazon: Amazonbot, Amzn-SearchBot, Amzn-User
- Mistral AI: Mistral crawlers
- DuckDuckGo: DuckAssistBot
- Common Crawl: CCBot