robots.txt syntax, rules and common mistakes

What is robots.txt?

robots.txt is a text file in the top-level directory of a website. It tells crawlers which URLs they may fetch. Google names its main purpose as avoiding too many requests to your site. In practice it also keeps crawlers away from areas that add nothing to search, such as the cart or internal search.

The rules follow the Robots Exclusion Protocol. Martijn Koster designed it in 1994. Since September 2022 it has been described in RFC 9309 from the IETF. Google follows it and adds details in its own documentation, for example on how it handles server errors.

A short file looks like this.

Does robots.txt control crawling or indexing?

The file controls crawling, meaning the fetch. Indexing is a separate step. A blocked URL can still end up in Google's index if other pages link to it. The result then appears without a description. The Google indexing guide has more on this.

noindex works the other way round. Google has to fetch the page to see the directive. If you block a page in robots.txt and add noindex as well, the directive stays invisible. The right tool depends on your goal.

How is robots.txt structured?

Each line contains a field, a colon and a value. Google evaluates four fields.

Field names are case-insensitive, paths are case-sensitive. Paths start with a slash, comments with #. Google does not evaluate fields such as crawl-delay and noindex in robots.txt, and it ignores invalid lines.

The wildcards * and $

The asterisk stands for any number of characters, including none. The dollar sign marks the end of the URL. Both characters are part of RFC 9309. Each rule applies from the start of the path and matches everything that begins that way. The examples show what this means.

The last row is a classic trap. If a parameter is not in first place, the rule does not apply. /*sort= covers both cases.

Which rules does a crawler follow?

A group starts with one or more User-agent lines, and the rules follow below them. A crawler looks for the group with its token, and case makes no difference. If there is none, the group with the asterisk applies. If that is missing too, the crawler may fetch everything. If the file has several groups for the same crawler, it merges them. Their order does not matter.

A crawler's own group replaces the asterisk group completely.

In this example Googlebot follows only its own group. It may fetch /cart/ and /account/, and only /internal/ is blocked for it. If the general blocks should apply to it as well, they have to be repeated in its group. The same goes for every AI crawler with its own group. Google also names one exception. AdsBot is not covered by the asterisk and has to be named explicitly.

Which rule wins?

Within a group, the order of the lines does not matter. The rule with the longest matching path wins. If an Allow and a Disallow rule are the same length, Allow applies. If no rule matches, the URL is allowed. The /robots.txt file itself can always be fetched.

Where does the file live and how big can it be?

• Location. Crawlers read only /robots.txt in the top-level directory. They do not check subfolders, and the file name is lowercase.
• Scope. The rules apply to exactly one host, one protocol and one port. The file on https://www.example.com does not apply to https://example.com, https://shop.example.com or http://www.example.com. Every subdomain needs its own file.
• Size. According to RFC 9309, crawlers must process at least 500 KiB. Google reads the first 500 KiB and ignores the rest.
• Format. The file is plain text in UTF-8. Google ignores a BOM, and characters outside UTF-8 can invalidate rules.
• Cache. According to RFC 9309, crawlers should cache the file for no more than 24 hours. Google generally does the same. The Cache-Control value max-age can change the duration, and during timeouts or server errors the copy stays in use for longer.

Which status codes apply to robots.txt?

The status code of /robots.txt decides whether a crawler gets going at all. RFC 9309 sets the framework, and Google describes its behaviour in more detail.

So a 404 is harmless, whereas a 503 on robots.txt is a real problem. If you deliver the file through a CDN or firewall, keep an eye on this response too. By the way, 401 and 403 will not slow crawlers down, because Google treats both codes like a missing file.

How do robots.txt and sitemaps fit together?

robots.txt may point to sitemaps. The line stands independently of the groups and therefore applies to all crawlers. It contains an absolute URL, several lines are allowed, and the sitemap may live on another host.

A sitemap entry does not lift a block. If a URL is in the sitemap and blocked by Disallow, it stays blocked for the crawler. The XML sitemap guide covers structure and limits.

robots.txt for AI crawlers

AI crawlers read the same file as search engines and follow the same groups. What matters is that operators offer separate tokens for each purpose. At OpenAI, GPTBot collects content that may be used for training. OAI-SearchBot makes pages appear in ChatGPT search. According to OpenAI, each setting is independent of the other. At Anthropic, ClaudeBot handles training and Claude-SearchBot handles search.

Fetches on user request work differently. According to their operators, ChatGPT-User, Perplexity-User and meta-externalfetcher do not reliably follow robots.txt, because a person triggers the fetch. According to Anthropic, Claude-User does follow it.

All bots, their purposes and what a block does in each case are covered in the AI crawlers guide and the AI bot list. The Managing AI bots use case shows how to handle this day to day.

Examples for typical cases

The first three examples cover the basic cases. The first one has the same effect as a missing file and is only here for completeness.

Each block stands for a separate file. A shop could structure its file like this.

Common mistakes

• The test environment's block goes live. A Disallow: / slips onto the production site during a release. The staging environment guide shows how to avoid this.
• Disallow instead of noindex. The page stays in the index, and Google can no longer read a noindex on it.
• CSS or JavaScript blocked. You may block unimportant resources. If Google needs the files to understand the page, they have to stay accessible.
• General rules missing from a bot's group. With its own group, the blocks from the asterisk group no longer apply to that bot.
• File on the wrong host. After a move, or on a new subdomain, the file is missing where it should apply.
• Server error on /robots.txt. A 503 stops crawling of the whole site, even if all other URLs work.
• HTML instead of text. If /robots.txt returns an HTML page with status 200, the lines are invalid. Google ignores them, and not a single rule applies.
• Wrong case. Disallow: /Shop/ does not block /shop/.
• Confidential paths listed. The file is public, so an entry shows the path to anyone who looks.

Testing and monitoring robots.txt

The robots.txt report in Search Console shows which files Google has found, when it last fetched them and which errors it found while reading them. It covers the 20 hosts with the highest crawl rate in a Domain property and shows fetches from the last 30 days. After an urgent change, you can request a new fetch there.

You can test individual URLs in the free robots.txt Tester, including for GPTBot, ClaudeBot and other bots. Developers can also run Google's open-source library locally. Because a release can change the file at any time, ongoing robots.txt monitoring is worth adding.

FAQ