- In March 2026 the source code of Claude Code was accidentally published via npm. In June 2026 we analyzed how the tool searches for web pages, fetches them and passes them on to the language model.
- The model picks search results by title and URL only. The page is converted to Markdown, scripts including JSON-LD are dropped, and a smaller model summarizes the rest before Claude sees it.
- For GEO this means the title has to earn the fetch, and the most important facts should be visible text in the delivered HTML.
What is AI grounding?
Grounding means that a language model ties its answer to sources it has just fetched. ChatGPT, Perplexity, Gemini or Claude search the web, load individual pages and write an answer from them. Hardly any provider discloses how this works in detail. GEO therefore mostly relies on observations and tests.
The source code of Claude Code shows such a pipeline step by step. It applies to this one tool, a program for developers in the terminal. Whether Claude.ai or other systems work the same way cannot be derived from it. The basic patterns still help you understand AI visibility better.
What happened in the leak?
On 31 March 2026 a release of Claude Code contained a source map file. It pointed to the unobfuscated TypeScript code, according to The Register around 1,900 files with more than 512,000 lines. Anthropic called it a packaging error caused by human error. According to heise, the affected version was removed from npm. By then copies were already circulating.
Four files matter for the analysis. WebSearchTool.ts handles search, WebFetchTool/utils.ts the fetch, WebFetchTool/prompt.ts the instruction to the summarization model, and preapproved.ts a list of preferred domains.
The pipeline in five stages
Every page that Claude Code fetches passes through five stages. Information is lost at each one. The values come from the code, as of March 2026.
| Stage | What happens | What is lost |
|---|---|---|
| 1 Search | The model sees the list of results and decides which pages to open | Snippet, meta description and date. Title and URL remain |
| 2 Fetch | The page is loaded, 10 MB at most, with a 15-minute cache | Everything that only JavaScript generates |
| 3 Conversion | The Turndown library turns HTML into Markdown, with default rules | head, script tags including JSON-LD, all HTML attributes |
| 4 Summary | A smaller model (Claude Haiku) answers the question about the page | The exact wording. Quotes are limited to 125 characters |
| 5 Main model | Claude works with the result of stage 4 | Everything the summary left out |
Title and URL decide whether a page is fetched
Before the selection, the search removes fields such as page_age. The model therefore decides on the basis of around 60 to 100 characters whether to open your page at all. A good meta description does not help at this point.
A second model reads your page first
After the fetch, the content goes to Claude Haiku, together with the main model's question. The instruction in the code is clear. Quotes may be at most 125 characters long. Haiku should render everything else in its own words. What Claude knows about your page is therefore a retelling.
89 domains get the original text
The file preapproved.ts lists 89 domains, almost all technical documentation such as MDN, react.dev, docs.python.org and kubernetes.io. According to our analysis, for these the code skips the summary and passes on up to 100,000 characters of original text. For a developer tool that makes sense. All other pages, meaning shops, blogs and company websites, only arrive in summarized form.
What gets through, what is lost?
| Lost in the conversion | Reaches the model |
|---|---|
| Meta description, Open Graph tags | Headings, paragraphs, lists |
| JSON-LD, microdata and RDFa attributes | Tables and code blocks |
| Canonical and other link elements in the head | Link texts and URLs in the content |
| Content that is only created by JavaScript | Navigation and footer, mixed in with the content |
| CSS classes, IDs, data attributes | Hidden elements with display:none or aria-hidden |
The last row is a trap. Turndown reads the HTML source and does not recognize hidden elements. Old promotional banners, hidden mobile menus or expired A/B variants therefore end up in the Markdown too and compete with your actual content. This is what it looks like in the code.
This matches our own test from October 2025. When fetching the page directly, none of the AI systems tested found a price that was only in the JSON-LD. The details are in the guide Schema markup and AI.
What has changed since?
Anthropic now describes the behavior itself in the Claude Code documentation (retrieved on 15 October 2026). WebFetch converts HTML into Markdown and, for most fetches, has a separate model evaluate the content. Claude only gets the result of this evaluation. The documentation itself calls this “lossy by design”. Also confirmed are 100,000 characters per fetch, a 15-minute cache and a built-in list of pre-approved documentation domains. According to the documentation, search only returns titles and URLs of the results.
Two points are new and useful for website owners. WebFetch identifies itself with a user agent that starts with Claude-User. This lets you find the fetches in your log files. WebFetch also asks for Markdown first via the Accept header. If your server delivers Markdown on request, the conversion is skipped. The documentation does not say whether the summary still applies in that case. Haiku, the 125 characters and the 10 MB are not mentioned there either. They remain findings from the code.
How do ChatGPT, Perplexity, Gemini and AI Overviews work?
Each system works differently. Google writes that AI Overviews and AI Mode build on the normal index and the ranking systems. A page has to be indexed and eligible to appear as a snippet. Structured data is not a requirement for this. OpenAI separates OAI-SearchBot for search in ChatGPT from ChatGPT-User for fetches triggered by a user. How these systems process fetched pages further is not disclosed. The conclusions below therefore only apply with certainty to Claude Code. They do match what we see when comparing the systems. Find out more in the guide GEO vs. SEO.
Five measures for your pages
- Write the title and URL with a clear topic. The model often decides on title and URL alone whether to fetch the page. State specifically which question the page answers.
- Put the most important part at the top. A summary under time pressure is more likely to leave out details from the middle. Start with the result.
- Move facts from the markup into the text. Price, delivery time, rating, date and author belong visibly on the page, in addition to the structured markup.
- Deliver important content server-side. Claude Code does not run JavaScript. Whatever is only created in the browser does not arrive there.
- Remove hidden content. Delete old banners and variants from the HTML. Hidden with CSS, they stay in the source code.
Common mistakes
- The title is a brand formula without a topic, such as “Welcome | Example Ltd”.
- Prices and opening hours are only in the JSON-LD or in a widget.
- The key message only comes after a long introduction, in the seventh paragraph.
- Bot protection or the robots.txt block Claude-User and other fetch bots without anyone noticing.
Frequently asked questions
What does grounding mean in AI?
Grounding means that a language model ties its answer to sources it has fetched for the question, such as web pages from a search. This lowers the risk of invented details and is the basis for AI answers citing websites.
Does Claude read my meta description?
According to the code and documentation of Claude Code, no. Search passes on title and URL, and the conversion to Markdown drops the head with all meta tags.
Does Claude see my schema markup?
Claude Code does not. JSON-LD sits in a script tag and is removed during the conversion. Facts that are meant to count therefore also belong in the visible text.
How do I recognize fetches by Claude in my log files?
WebFetch in Claude Code uses a user agent that starts with Claude-User. Anthropic also distinguishes ClaudeBot for training and Claude-SearchBot for search. Both can be controlled separately in the robots.txt.
Can I get onto the list of preferred domains?
There is no known procedure for this. The list is hard-coded and consists almost entirely of technical documentation.
Does this also apply to ChatGPT and Gemini?
Not necessarily. Each system processes pages differently, and most do not disclose how. Visible text in the delivered HTML and clear titles do help with every system that fetches pages live.
Sources
Retrieved on October 15, 2026. If something is out of date or missing, let us know.
- Claude Code Docs, Tools reference (WebFetch, WebSearch): https://code.claude.com/docs/en/tools-reference · Retrieved 15 Oct 2026
- Anthropic, crawlers and blocking (ClaudeBot, Claude-User, Claude-SearchBot): https://support.claude.com/en/articles/8896518
- The Register, Anthropic Claude Code source code: https://www.theregister.com/2026/03/31/anthropic_claude_code_source_code/ · 31 Mar 2026
- heise, Claude Code: Source Map legt Quellcode offen: https://www.heise.de/-11242079 · 1 Apr 2026
- Turndown, HTML-to-Markdown library: https://github.com/mixmark-io/turndown
- Google Search Central, Google's Guide to Optimizing for Generative AI Features on Google Search: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide · As of 10 Jul 2026
- Google Search Central, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features · As of 10 Dec 2025
- OpenAI, Crawler: https://developers.openai.com/api/docs/bots