Which AI crawlers should you allow?

What is an AI crawler?

An AI crawler is a bot that fetches web pages for an AI system. It works like a search engine crawler and identifies itself with its own user agent. What matters is what the operator uses the content for. That determines what a block in robots.txt does.

For fetches, robots.txt only applies to a limited extent. OpenAI writes that robots.txt rules may not apply to ChatGPT-User, because a human triggers the fetch. Perplexity and Meta say the same about their fetch bots. Anthropic, by contrast, describes Claude-User as controllable.

The key AI crawlers at a glance

Training crawls come mainly from GPTBot by OpenAI, ClaudeBot by Anthropic, meta-externalagent by Meta, Amazonbot, Bytespider and CCBot by Common Crawl. For AI search, OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot are on the move. At a user's request, ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher fetch individual pages. Googlebot, Bingbot and Applebot feed the search engines and their AI answers. They render JavaScript, most other AI crawlers do not.

All 39 bots and tokens with purpose, JavaScript support, robots.txt behavior, traffic and sources are in the filterable overview AI crawlers and JavaScript rendering. More tokens and the operators' IP lists are in the list of AI bots.

Google-Extended and Applebot-Extended are not crawlers of their own. They only control what Google and Apple may use the data from Googlebot and Applebot for.

Google-Extended and AI Overviews

Google-Extended is a token without a crawler of its own. It controls whether Google uses content to train future Gemini models and for grounding in the Gemini apps and on Vertex AI. According to Google, it affects neither inclusion in Google Search nor ranking. AI Overviews and AI Mode are governed by Googlebot. If you want to limit what appears there from your pages, use nosnippet, data-nosnippet, max-snippet or noindex.

Blocking or allowing AI crawlers

Many websites want to be mentioned in AI answers but do not want to hand over their content for training. This is possible because the major providers control training and search through separate tokens. Here is what a robots.txt looks like that excludes training and allows AI search.

According to RFC 9309, several User-agent lines above one rule are allowed. Each bot only follows the group with its own token. The rules under User-agent * then no longer apply to it. That is why the Disallow for /checkout/ also appears in the AI search group. GPTBot and the other training bots are blocked completely anyway. The fetch bots are missing from the example because, apart from Claude-User, none of them reliably follows the file.

Until a change takes effect

According to RFC 9309, crawlers may use a cached robots.txt for no longer than 24 hours. OpenAI, Perplexity and Meta give the same period. A block by IP address does not replace robots.txt. Anthropic points out that its bots then cannot even read robots.txt.

Do AI crawlers render JavaScript?

Most do not. For an analysis published in December 2024, Vercel tracked AI crawler traffic on its own network over several months. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, meta-externalagent, Bytespider, PerplexityBot and CCBot did not execute JavaScript. GPTBot and ClaudeBot do download JavaScript files (11.5% of requests for GPTBot, 23.8% for ClaudeBot), but they do not execute them.

Googlebot, Bingbot and Applebot render pages. Google uses an evergreen version of Chromium for this, and Bing has used Microsoft Edge since 2019. Apple writes that Applebot may render content in a browser. According to Vercel, Gemini relies on Googlebot's infrastructure. With Yandex, the site owner decides in Yandex Webmaster whether the bot executes JavaScript. For Claude-SearchBot, Claude-User, DuckAssistBot, YouBot and most fetch bots, the operators provide no information, and Vercel did not measure them.

How each individual bot handles JavaScript is shown in the guide AI crawlers and JavaScript rendering.

In our test from October 2025, Perplexity knew a price after indexing that only JavaScript had inserted. With one page, that is no proof. There is more on this in the guide Schema markup and AI.

For shops and single-page apps, this means product names, prices and text must be in the delivered HTML. Content that a script loads later is missing for most AI systems. Server-side rendering or pre-rendering the important pages solves this. Vercel's measurement dates from 2024. So use your own log files to check which bots fetch scripts and API endpoints today.

How much traffic AI crawlers cause

On Cloudflare's network, Googlebot accounted for 39% of all AI and search crawler traffic in July 2025. It was followed by GPTBot at 11.7%, ClaudeBot at 9.9%, Bingbot at 9.3% and meta-externalagent at 7.5%. Bytespider fell from 14.1% in July 2024 to 2.4%. In Cloudflare's 2025 Year in Review (1 January to 2 December), AI crawlers made up 20% of verified bot traffic and search crawlers 40%. Googlebot alone accounted for more than 28%, GPTBot for 7.5%.

Bot protection and CDN

Even if robots.txt allows a bot, the firewall or the CDN's bot protection can lock it out. In July 2025, Cloudflare announced that it would block AI crawlers by default unless they pay for content. From 15 September 2026, new defaults apply to new domains. AI search is allowed, training and agents are blocked on pages with ads. Crawlers that serve several purposes, such as Googlebot, Applebot or Bingbot, fall under the strictest rule. If you block training there, you block them too.

1. Read out the bot and AI settings of your CDN and your firewall, including the defaults that apply without you doing anything.
2. Compare them with your robots.txt. Both should want the same thing.
3. Check in the log files whether the bots you want to allow get status 200 and not a challenge page.
4. With rules against bots, it is worth checking whether they hit real visitors, for example via VPN. A graduated response with throttling blocks fewer of the wrong visitors than a hard block.

The guide Bot protection and SEO explains how to set up bot protection so that it slows down attackers and lets Googlebot, AI bots and customers through.

Telling real bots from fake ones

Anyone can set the user agent. OpenAI, Anthropic, Perplexity, Apple and Common Crawl therefore publish their IP ranges as a JSON list. Apple and Common Crawl also support verification via reverse DNS. A request with the GPTBot user agent from an IP outside OpenAI's list does not come from OpenAI.

Common mistakes

• Blocking all OpenAI bots when only training was meant to be excluded. The website then also disappears from ChatGPT search.
• Blocking Google-Extended to get out of AI Overviews. That is Googlebot's job.
• Overlooking new bots. Anthropic, for example, now has Claude-SearchBot and Claude-User in addition to ClaudeBot.
• robots.txt allows everything, but the CDN blocks anyway.
• Loading important content only via JavaScript.

FAQ