- Most AI crawlers only read the HTML the server delivers. Vercel measured no JavaScript execution for GPTBot, ClaudeBot, PerplexityBot and other major bots.
- Googlebot, Bingbot and Applebot render pages. According to Vercel, Gemini uses Googlebot's infrastructure, and ChatGPT agent works with its own browser.
- Anything that only appears in the browser is therefore missing for many AI systems. Prices, product texts and links belong in the delivered HTML.
Websites rely more and more on JavaScript. In a browser, nobody notices. A crawler only sees the same content if it renders the page, which means running the scripts the way a browser does. For search engines this has been standard for years. For AI crawlers it is the exception.
We first published this overview in November 2025 and re-checked it against the operators' documentation for the version of November 3, 2026. New entries include Google-Agent, meta-webindexer, Amazon's bots for search and fetching, and the three Mistral crawlers.
AI crawlers and JavaScript rendering at a glance
The list covers 39 bots and tokens that show up in log files and robots.txt files. You can filter it by JavaScript support and by purpose. The sources for each row are in the last column.
How to read the JavaScript column
| Label | Meaning |
|---|---|
| Renders JavaScript | The operator documents that the crawler renders pages, or a measurement shows it. |
| Partial | The crawler only renders under certain conditions, for example depending on a setting chosen by the site owner. |
| No JavaScript | Vercel measured in 2024 that the crawler does not execute scripts. Some bots download JavaScript files without executing them. |
| Not stated | The operator says nothing about it, and there is no published measurement. |
| No fetches of its own | Control tokens such as Google-Extended do not crawl. They only control what the data of another crawler is used for. |
| Bot and operator | Purpose | JavaScript | robots.txt | Traffic (as of) | Sources |
|---|---|---|---|---|---|
| GooglebotGoogle | Search engineGoogle Search including AI Overviews and AI Mode | yes, with an up-to-date Chromium | yes | 39% (07/2025) · 4.5 bn requests/month incl. Gemini (12/2024) | Google · Google · Vercel · Cloudflare |
| BingbotMicrosoft | Search engineBing search; Bing's AI answers also draw on this index | yes, with Microsoft Edge (since 2019) | yes, including crawl-delay | 9.3% (07/2025) | Microsoft · Cloudflare |
| ApplebotApple | Search engineSpotlight, Siri and Safari; the data may be used to train Apple models | yes, Apple says Applebot renders in a browser | yes, no crawl-delay; without its own rules it follows the Googlebot rules | 1.5% (07/2025) · 314 m requests/month (12/2024) | Apple · Vercel · Cloudflare |
| YandexBotYandex | Search engineYandex search | depends on the setting in Yandex Webmaster | yes | not reported | Yandex |
| GPTBotOpenAI | TrainingTraining OpenAI models | no, fetches JS files (11.5% of requests) without executing them | yes, per OpenAI a disallow means no training | 11.7% (07/2025) · 569 m requests/month (12/2024) · +305% requests (05/2024 to 05/2025) | OpenAI · Vercel · Cloudflare · Cloudflare |
| ClaudeBotAnthropic | TrainingTraining Claude models | no, fetches JS files (23.8% of requests) without executing them | yes, including crawl-delay | 9.9% (07/2025) · 370 m requests/month (12/2024) · −46% requests (05/2024 to 05/2025) | Anthropic · Vercel · Cloudflare · Cloudflare |
| meta-externalagentMeta | TrainingTraining Meta's AI models and improving Meta products | no | yes | 7.5% (07/2025) · 19% of AI-only crawler traffic (05/2025) | Meta · Vercel · Cloudflare · Cloudflare |
| AmazonbotAmazon | TrainingAmazon products; the data may be used to train Amazon models | not stated, not measured | yes, no crawl-delay | 5.9% (07/2025) · −35% requests (05/2024 to 05/2025) | Amazon · Cloudflare · Cloudflare |
| BytespiderByteDance | Trainingnot stated by the operator | no | not stated by the operator | 2.4% (07/2025) · −85% requests (05/2024 to 05/2025) | Vercel · Cloudflare · Cloudflare |
| CCBotCommon Crawl | TrainingOpen web archive, often used as a basis for training data | no, does not render pages according to Vercel | yes | not reported | Common Crawl · Vercel |
| MistralAI-TrainingMistral AI | TrainingDatasets for training Mistral models | not stated, not measured | yes, can be blocked with disallow | not reported | Mistral AI |
| OAI-SearchBotOpenAI | AI searchSources for search in ChatGPT | no | yes, changes take effect after about 24 hours | 0.9% (07/2025) | OpenAI · Vercel · Cloudflare |
| Claude-SearchBotAnthropic | AI searchSearch in Claude | not stated, not measured | yes | not reported | Anthropic |
| PerplexityBotPerplexity | AI searchSearch in Perplexity, no training according to Perplexity | no | yes, changes take effect within 24 hours | 24.4 m requests/month (12/2024) · +157,490% requests (05/2024 to 05/2025) | Perplexity · Vercel · Cloudflare |
| meta-webindexerMeta | AI searchSearch results in Meta AI | not stated, not measured | yes | not reported | Meta |
| Amzn-SearchBotAmazon | AI searchSearch, for example for Alexa, no training according to Amazon | not stated, not measured | yes; without its own rules it follows the rules for other search bots | not reported | Amazon |
| MistralAI-IndexMistral AI | AI searchSearch index for answers in Mistral Vibe, no training according to Mistral | not stated, not measured | not stated by the operator | not reported | Mistral AI |
| DuckAssistBotDuckDuckGo | AI searchAI answers with sources, no training according to DuckDuckGo | not stated, not measured | yes, a block takes effect after 72 hours | not reported | DuckDuckGo |
| YouBotYou.com | AI searchSearch index of You.com | not stated, not measured | yes, including crawl-delay | not reported | You.com |
| ChatGPT-UserOpenAI | User-triggered fetchFetch when someone asks for it in ChatGPT or a custom GPT | no | per OpenAI, the rules may not apply | 0.9% (07/2025) · +2,825% requests (05/2024 to 05/2025) | OpenAI · Vercel · Cloudflare · Cloudflare |
| Claude-UserAnthropic | User-triggered fetchFetch when someone asks Claude a question | not stated, not measured | yes, a block prevents the fetch | not reported | Anthropic |
| Perplexity-UserPerplexity | User-triggered fetchFetch when someone asks Perplexity a question | not stated, not measured | generally no | not reported | Perplexity |
| meta-externalfetcherMeta | User-triggered fetchFetches individual links at a user’s request | not stated, not measured | may bypass the rules | not reported | Meta |
| facebookexternalhitMeta | User-triggered fetchLink preview when someone shares a link in Meta's apps | not stated, not measured | may bypass the rules during security checks | not reported | Meta |
| Amzn-UserAmazon | User-triggered fetchLive fetch, for example for Alexa questions, no training according to Amazon | not stated, not measured | may not follow all rules, according to Amazon | not reported | Amazon |
| MistralAI-UserMistral AI | User-triggered fetchFetch when someone asks Mistral Vibe a question | not stated, not measured | yes, the token controls which pages may be fetched | not reported | Mistral AI |
| Google-GeminiNotebookGoogle | User-triggered fetchFetches URLs that users add as sources in Gemini Notebook (formerly Google-NotebookLM) | not stated, not measured | generally no | not reported | |
| ChatGPT agent (no token of its own)OpenAI | AgentCompletes tasks on behalf of a user; requests are signed | yes, works with its own browser according to OpenAI | not stated by the operator | not reported | OpenAI · OpenAI Help |
| Google-AgentGoogle | AgentAgents on Google infrastructure that browse the web and take actions for users | not stated by the operator | generally no | not reported | |
| Google-ExtendedGoogle | Control tokenControls training and grounding for Gemini and Vertex AI, with no effect on Google Search | no fetches of its own, the data comes from Googlebot | disallow excludes training and grounding | not reported | Google · Google |
| Applebot-ExtendedApple | Control tokenControls whether Apple uses Applebot's data for training | no fetches of its own, the data comes from Applebot | disallow excludes training, search stays | not reported | Apple |
| Gemini (no crawler of its own)Google | Control tokenUses Googlebot's infrastructure according to Vercel | yes, via Googlebot's infrastructure | controlled via Googlebot and Google-Extended | included in Googlebot's 4.5 bn requests (12/2024) | Vercel · Google |
| OAI-AdsBotOpenAI | OtherChecks pages submitted as ads in ChatGPT | not stated, not measured | not stated, only visits submitted pages | not reported | OpenAI |
| meta-externaladsMeta | OtherMeta's advertising and business products | not stated, not measured | yes | not reported | Meta |
| Google-CloudVertexBotGoogle | OtherCrawls at the site owner’s request for agents in Vertex AI | not stated, not measured | yes | not reported | |
| Claude-WebAnthropic | OutdatedOlder Anthropic token, no longer listed in the current docs | not stated | no longer documented | not reported | Anthropic |
| anthropic-aiAnthropic | OutdatedOlder Anthropic token, no longer listed in the current docs | not stated | no longer documented | not reported | Anthropic |
| FacebookBotMeta | OutdatedOlder token, no longer listed in Meta's current docs | not stated | no longer documented | not reported | Meta |
| cohere-aiCohere | OutdatedToken for which we found no documentation from Cohere | not stated | no operator documentation found | not reported | no primary source found |
| Showing: | |||||
The percentages in the traffic column are shares of all AI and search crawler traffic on Cloudflare's network in July 2025. The changes compare requests in May 2025 with May 2024, also according to Cloudflare. Vercel measured the requests per month on its own network and published them in December 2024. Tokens marked as outdated are no longer documented by their operator, or we found no documentation for them. They still appear in old robots.txt files and log files.
How training, search and user-triggered fetches fit together, and what a robots.txt for them looks like, is explained in the guide AI crawlers. More tokens and the operators' IP lists are in the list of AI bots.
Why is JavaScript a problem for AI crawlers?
For 14 entries in the overview there is an operator statement or a measurement. 8 of them do not execute JavaScript, including GPTBot, ClaudeBot and meta-externalagent. Together these three accounted for 29% of crawler traffic on Cloudflare in July 2025. For its analysis from December 2024, Vercel tracked their requests over several months. GPTBot fetched JavaScript files in 11.5% of its requests, ClaudeBot in 23.8%. Neither of them executed the scripts.
An AI crawler without rendering only knows the content in the delivered HTML. If a shop loads product lists, prices or reviews in the browser, GPTBot gets an almost empty page. This affects the training data of future models and AI search alike. OAI-SearchBot and PerplexityBot supply the sources for ChatGPT and Perplexity, and according to Vercel they do not render JavaScript either.
Vercel measured something else as well. For ChatGPT, 34.8% of fetches ended on 404 pages, for Claude 34.2% and for Googlebot 8.2%. Clean redirects and up-to-date sitemaps therefore help AI crawlers too.
How have AI crawlers developed?
The timeline shows the key steps since GPTBot launched. The information comes from operator announcements and from measurements by Vercel and Cloudflare.
| Date | What happened |
|---|---|
| 2023 | OpenAI documents GPTBot. Since then, site owners can exclude training of OpenAI models via robots.txt. |
| September 28, 2023 | Google introduces Google-Extended, a token without a crawler of its own. It first controlled use for Bard and Vertex AI, today for Gemini. |
| December 17, 2024 | Vercel publishes measurements from its own network. None of the major AI crawlers executes JavaScript. Gemini and Applebot render. |
| January 23, 2025 | OpenAI presents Operator, an agent with its own browser. |
| May 2025 | According to Cloudflare, GPTBot reaches 30% of AI-only crawler traffic, up from 5% in May 2024. Its requests grow by 305%. meta-externalagent appears for the first time and reaches 19%. |
| July 17, 2025 | OpenAI launches ChatGPT agent. Operator becomes part of it. The agent still works with its own browser. |
| July 2025 | On Cloudflare, Googlebot accounts for 39% of all AI and search crawler traffic, GPTBot for 11.7% and ClaudeBot for 9.9%. |
| 2025 and 2026 | Anthropic, Meta, Amazon and Mistral split training, search and fetching into separate tokens. Google documents Google-Agent for agents that browse the web on behalf of users. |
The most important AI crawlers in detail
GPTBot
GPTBot collects content for training OpenAI's models. Vercel counted around 569 million requests per month in December 2024. On Cloudflare its share was 11.7% of all AI and search crawlers in July 2025. GPTBot downloads JavaScript files but does not execute them. Search in ChatGPT is handled by OAI-SearchBot, which does not render either.
ClaudeBot
ClaudeBot collects training data for Claude. According to Vercel, 23.8% of its requests were JavaScript files. It did not execute them. Vercel counted around 370 million requests per month, and Cloudflare a share of 9.9% in July 2025. For search and user-triggered fetches, Anthropic now uses Claude-SearchBot and Claude-User.
Googlebot
Googlebot renders pages with an up-to-date version of Chromium. It feeds Google Search and therefore AI Overviews and AI Mode. At 39%, it had the largest share of crawler traffic on Cloudflare in July 2025. According to Vercel, Gemini uses the same infrastructure and therefore also sees content that JavaScript creates.
meta-externalagent
meta-externalagent crawls to train Meta's AI models and to improve Meta products. According to Cloudflare it reached 19% of AI-only crawler traffic in May 2025 and 7.5% of all AI and search crawlers in July 2025. Vercel measured no JavaScript execution. For search in Meta AI, Meta now uses meta-webindexer, and meta-externalfetcher for user-triggered fetches.
Bingbot
Bingbot has rendered with Microsoft Edge since October 2019 and supports crawl-delay. In July 2025 it accounted for 9.3% of crawler traffic on Cloudflare. Bing's AI answers also draw on the Bing index.
PerplexityBot
PerplexityBot builds Perplexity's search index, and according to Perplexity it is not used for training. Between May 2024 and May 2025 its requests on Cloudflare grew by 157,490%, from a very small base. According to Vercel it does not execute JavaScript. User-triggered fetches run through Perplexity-User, which generally ignores robots.txt.
How to check what AI crawlers see
The quickest way is to compare the delivered HTML with the rendered page. If content only appears in the rendered page, most AI crawlers miss it.
- Open the page source (view-source) or load the page with JavaScript switched off. If product names, prices or texts are missing, a script loads them later.
- Compare source code and DOM with the free JavaScript Rendering Check. It shows which content only appears through JavaScript.
- Fetch the HTML without rendering on the command line and search it for the content you want to show up in AI answers.
- Check your log files. They show which AI bots fetch your pages, whether they get status 200, and whether they also load scripts and API endpoints.
If grep finds the price in the delivered HTML, GPTBot and ClaudeBot can read it too. If it is missing, only a crawler that renders will see it. In our test from October 2025, Perplexity still knew a price after indexing that only JavaScript had inserted. One page is no proof. There is more on this in the guide Schema markup and AI.
What can you do?
Render important content on the server
Server-side rendering (SSR) or pre-rendering delivers the finished content in the HTML. Frameworks such as Next.js or Nuxt support this out of the box. Vercel recommends delivering at least the main text, metadata and navigation from the server. Interactive elements can still be built in the browser.
HTML first, JavaScript as an enhancement
With progressive enhancement the content sits in the HTML, and JavaScript adds filters, the basket or animations. If a crawler does not run the script, the content stays readable. Semantic HTML with clear headings helps as well.
Structured data and meta tags in the HTML
JSON-LD that only a script inserts is missing for crawlers without rendering. So output structured data, title, canonical and robots tags directly in the delivered HTML.
Keep checking rendering and bots
A release can quietly move content from the server to the browser. Compare source code and rendered page regularly, and check in your log files which bots fetch which pages. The list of AI crawlers keeps changing as well.
What comes next?
AI agents such as ChatGPT agent work with a browser and see pages the way users do. For the training and search crawlers, no major operator has so far documented that they render. At the same time, operators keep splitting their bots more finely by purpose. For AI visibility you therefore need both, HTML that contains all important content and a robots.txt that matches the current tokens. How to prepare your website for agents is covered in the guide Agentic Engine Optimization.
Frequently asked questions
Can ChatGPT read JavaScript?
According to Vercel's measurements, OpenAI's crawlers for training (GPTBot), search (OAI-SearchBot) and user-triggered fetches (ChatGPT-User) do not execute JavaScript. ChatGPT agent, on the other hand, works with its own browser and sees pages like a user.
Does Gemini render JavaScript?
Gemini has no crawler of its own. According to Vercel, Gemini uses Googlebot's infrastructure, which renders pages with Chromium. Whether Google may use your content for Gemini is controlled by the Google-Extended token.
Does Perplexity see content loaded with JavaScript?
According to Vercel, PerplexityBot does not execute JavaScript. In our test from October 2025, a price inserted by JavaScript still appeared in an answer. You should not rely on that. Important content belongs in the delivered HTML.
Is it enough if Googlebot renders my page?
For Google Search with AI Overviews and AI Mode, yes, because Googlebot is the crawler there. ChatGPT, Claude and Perplexity use their own crawlers, which as far as is known only read the delivered HTML.
Do AI crawlers download my JavaScript files?
Some do. According to Vercel, GPTBot and ClaudeBot fetch JavaScript files without executing them. Your log files show which bots load scripts and API endpoints.
Sources
Retrieved on November 3, 2026. If something is out of date or missing, let us know.
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- OpenAI, Introducing Operator: https://openai.com/index/introducing-operator/ · January 23, 2025
- OpenAI, Introducing ChatGPT agent: https://openai.com/index/introducing-chatgpt-agent/ · July 17, 2025
- Anthropic, Does Anthropic crawl data from the web?: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Google, Google's common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers · as of November 3, 2026
- Google, Google's user-triggered fetchers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers · as of November 3, 2026
- Google Search Central, Understand JavaScript SEO basics: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
- Google, An update on web publisher controls: https://blog.google/technology/ai/an-update-on-web-publisher-controls/ · September 28, 2023
- Apple, About Applebot: https://support.apple.com/en-us/119829
- Microsoft Bing, The new evergreen Bingbot: https://blogs.bing.com/webmaster/october-2019/the-new-evergreen-bingbot-simplifying-seo-by-leveraging-microsoft-edge · October 9, 2019
- Meta, Web Crawlers: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Amazon, Amazonbot: https://developer.amazon.com/amazonbot
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/guides/bots
- Mistral AI, Mistral crawlers: https://docs.mistral.ai/robots
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- You.com, YouBot: https://you.com/docs/youbot
- Yandex Webmaster, JavaScript page rendering: https://yandex.com/support/webmaster/en/yandex-indexing/rendering
- Common Crawl, CCBot: https://commoncrawl.org/ccbot
- Vercel, The rise of the AI crawler: https://vercel.com/blog/the-rise-of-the-ai-crawler · December 17, 2024
- Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025: https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025 · July 1, 2025
- Cloudflare, The crawl-to-click gap: https://blog.cloudflare.com/crawlers-click-ai-bots-training/ · August 29, 2025