llms.txt explained, with structure, example and limits

What is llms.txt?

llms.txt is a text file in Markdown format at /llms.txt. It summarizes what a website is about and links to the pages a language model or AI agent should know. Jeremy Howard proposed it in September 2024. The description is on llmstxt.org. The current version 2 dates from 10 August 2026.

The proposal addresses a practical problem. HTML pages contain navigation, scripts and ads, and a model has limited room in its context. A short overview in Markdown is meant to do the selection for it.

What does the file not control?

llms.txt does not control access. Which bots may fetch your pages is still set in robots.txt. That is also where you opt out of AI model training, for example with dedicated rules for GPTBot.

Structure and example

The specification defines a fixed order. Only the first line, with the name of the website as a heading, is required.

This is what an llms.txt for a fictional shop looks like.

llms-full.txt and Markdown versions

Many generators and documentation platforms also create an llms-full.txt. It contains the complete content in a single file. It does not appear in the specification. Instead, the specification recommends Markdown versions of individual pages at the same URL, with .md appended or replacing the extension.

Files in subdirectories

Since version 2, an llms.txt may also sit in a subdirectory, for example at /docs/llms.txt for the /docs/ section. If there are several, the most specific one applies. Chrome Lighthouse, however, only checks the file in the root of the domain.

Who reads llms.txt?

Whether the file achieves anything depends on who fetches and uses it. The table shows the public statements as of 15 October 2026.

Publishing your own file is not the same as evaluating other websites' files.

What do studies measure?

In May 2026, Ahrefs analyzed around 137,000 domains with traffic. 28% had an llms.txt, and 97% of these files were not fetched a single time during the whole month. Among AI bots, GPTBot fetched the files most often, followed by Claude Code. A fetch does not show, however, whether the content feeds into answers.

In September 2026, Rankability found a valid llms.txt on 9.3% of the 1,000 most visited domains. The figures differ widely because the studies count different domains and check with different strictness.

Will an llms.txt get you mentioned more often in AI answers?

That is the hope behind most of these files. So far there is no evidence for it. OpenAI, Anthropic, Perplexity and Google do not say that they evaluate other websites' llms.txt for their answers. Google states explicitly that no such file is needed for AI Overviews and AI Mode.

The access figures point the same way. In the Ahrefs study, only 1.1% of llms.txt fetches came from bots that retrieve content for live answers. Overall, 96% of fetches came from bots, most of them with no recognizable link to AI answers.

llms.txt, robots.txt and XML sitemap

Do you need an llms.txt?

Create and check an llms.txt

1. Decide which pages a model should know, such as products, docs, pricing and contact.
2. Write the file in Markdown following the structure above and save it as /llms.txt in the root.
3. Check that /llms.txt is served as text with status 200.
4. Open every link once. All of them should return status 200 and must not be blocked for AI bots in robots.txt.
5. Test the page with Chrome Lighthouse in the “Agentic Browsing” category.
6. Decide who updates the file when the structure or URLs change.

Common mistakes

• /llms.txt returns HTML because the CMS serves an error page or the homepage.
• A catch-all rule returns status 200 even though no file exists.
• Links point to redirects, deleted pages or URLs that robots.txt blocks for AI bots.
• After a relaunch, the file still points to the old URLs.

FAQ