robots.txt Tester Test your robots.txt for Google and AI bots

The tester reads your robots.txt and shows for every URL which bot may fetch it. Change rules in the editor and test again straight away.

Fetched as searchVIUbot.

We store the request with your IP address to run the tool. More in the privacy policy.
How Google evaluates it

The longest matching rule wins

The tester follows the rules Google documents for robots.txt. If an Allow and a Disallow rule are the same length, Allow applies.

CaseWhat Google does
Group for the botGoogle uses the group with the matching user-agent. If there is none, the group for all bots (*) applies.
Wildcards* stands for any characters, $ for the end of the URL.
robots.txt missing (4xx except 429)Google crawls without restrictions.
Server errors (5xx) and 429Google stops crawling the website for up to 12 hours. After that, Google uses the last fetched robots.txt for up to 30 days.
SizeGoogle reads the first 500 KiB and ignores the rest.

Source: Google Search Central, “How Google interprets the robots.txt specification”

All AI bots with their robots.txt tokens
AI bots

Every AI bot has its own token

OpenAI, Anthropic and Perplexity send several bots with different jobs. If you want to control them individually, each one needs its own rule.

  • TrainingGPTBot, ClaudeBot and CCBot collect content for AI models.
  • AI searchOAI-SearchBot and PerplexityBot fetch pages for answers with sources.
  • Google-ExtendedA robots.txt token for Gemini only, with no crawler of its own.
  • GooglebotGoogle-Extended doesn't change how Google crawls your pages for search.
Controlling AI bots: the use case
robots.txtExample
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /warenkorb/
GPTBot blockedOAI-SearchBot allowedGooglebot allowed
Resources

Blocked files are missing when the page renders

If robots.txt blocks JavaScript or CSS files, Google sees the page differently from a browser. If you like, the tester checks every file your homepage loads.

JavaScriptIf scripts are blocked, content and links that only appear through JavaScript are missing.
CSSWithout stylesheets, Google is less able to judge layout and mobile rendering.
ImagesBlocked images don't appear in Google Images.

Files on the robots.txt host are checked. Files on other hosts follow those hosts' own robots.txt.

JavaScript SEO and rendering

Frequently asked questions

How is the tester different from robots.txt Monitoring?
The tester shows the state at the moment you check. The monitoring checks the file every 15 minutes and reports every change, even if someone else deploys it.
Does the editor change my robots.txt?
No. You test a copy in your browser. The file on your server stays as it is.
Which user agent does the tester use to fetch the file?
searchVIUbot. The tester then evaluates the rules for each bot separately, as Google documents it.
Why does a URL say “other host”?
A robots.txt only applies to the host it sits on. www.example.com and shop.example.com each need their own file.
How many URLs can I test?
Up to 100 URLs or paths per check. After that, you can test again in the editor as often as you like.
What do you store from my request?
We store the addresses you entered, the result and your IP address to run the tool. We don't store page content. We also use the checked domain to see which companies are interested in the topic.
Demo

What does your website look like in searchVIU?

After a short call about crawler access and Search Console, Michael shows you searchVIU with your website and a demo project. The demo takes around 30 minutes.

Request a demo