llms.txt Checker Check your llms.txt or create one straight away

The checker reads your llms.txt and checks structure, links and robots.txt blocks for AI bots. If there's no file, it creates a draft from your sitemap.

Fetched as searchVIUbot.

We store the request with your IP address to run the tool. More in the privacy policy.
Structure

A title, then sections with links

The llmstxt.org proposal describes a Markdown file at /llms.txt. Only the title is required. A short description and sections help AI systems put the content in context.

What an llms.txt looks like

# Example Ltd

> Manufacturer of measurement technology for industry.

## Products

- [Sensors](https://www.example.com/sensors/): Overview of all product lines
- [Software](https://www.example.com/software/): Analysis and interfaces

## Optional

- [Press](https://www.example.com/press/)

What the checker tests

Title with #, short description with >, sections with ## and link entries in the format - [Name](URL): Note.

It also flags invalid, relative and duplicate URLs, links to other hosts and the file's content type.

If there is also an llms-full.txt, the checker shows whether it is reachable and how large it is.

Source: llmstxt.org, proposal by Jeremy Howard (2024)

llms.txt in the guide
AI bots

llms.txt and robots.txt have to match

If robots.txt blocks GPTBot or ClaudeBot, those bots can't fetch the pages in your llms.txt either. The checker compares both for every linked URL.

Example rules: Disallow: /products/ for GPTBot, Disallow: /pricing/ for all bots.

  • GPTBot and OAI-SearchBotOpenAI's bots for training and search.
  • ClaudeBotAnthropic's bot.
  • PerplexityBotPerplexity's bot for search.
  • The file itself/llms.txt can also be blocked by robots.txt.
Controlling AI bots: the use case
Linked URLsSchematic example
/products/blocked: GPTBot
/pricing/blocked: all bots
/guides/allowed
Context

Whether AI systems read the file is still open

So far, llms.txt is a proposal from the community. Your server logs show which AI bots actually fetch it.

Create the fileThe draft from your sitemap is a good starting point.
Measure requestsLog file analysis shows which bots request /llms.txt and how often.
Keep it up to dateLinked pages change. llms.txt Monitoring checks the links every day.
See AI log file analysis

Frequently asked questions

Does llms.txt replace robots.txt?
No. robots.txt controls who may fetch what. llms.txt describes content for AI systems and doesn't block anything.
What is llms-full.txt?
A variant that includes the content of the pages, usually as one large Markdown file. The checker shows whether it exists and how large it is.
How is the draft created?
The checker takes your sitemap, picks up to 25 pages from different sections and uses their title and meta description. Pages with noindex are left out. Review the selection and the texts before you publish the file.
How many links does the checker test?
It reads every link in the file. It fetches the first 50 and checks the robots.txt rules for the first 150.
Which AI bots does the checker test?
GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic and PerplexityBot. You'll find every bot with its token in the AI bot list.
What do you store from my request?
We store the URLs you entered, the result and your IP address to run the tool. Fetched page content is deleted automatically after seven days. We also use the checked domain to see which companies are interested in the topic.
Demo

What does your website look like in searchVIU?

After a short call about crawler access and Search Console, Michael shows you searchVIU with your website and a demo project. The demo takes around 30 minutes.

Request a demo