AI Crawler Monitoring When one line locks out AI bots, searchVIU notices.
searchVIU checks your robots.txt every 15 minutes for search engines and AI bots. If a change locks out a bot, you see the affected URLs and their Google clicks.
GooglebotSearchOAI-SearchBotAI searchClaude-SearchBotAI searchPerplexityBotAI searchChatGPT-UserFetch on user requestGPTBotAI training · own groupEvery AI bot has its purpose, from training and search to answers.
OpenAI sends GPTBot for training, OAI-SearchBot for search and ChatGPT-User when someone asks a question in ChatGPT. Each bot has its own setting. A rule for all bots, however, also hits the bots that fetch for search.
Which bot is allowed in, and what does a block hit?
searchVIU assigns every known bot to a class, from search, AI search, AI training and fetches on user request to SEO tools. For each bot you see whether it is allowed in, whether it has its own group and how many URLs a block hits. That way you can block training and let AI search in.
| Bot | Class | Access | Blocked URLs | Google clicks on these URLs |
|---|---|---|---|---|
Googlebot | Search | allowed | 0 | 0 |
OAI-SearchBot | AI search | blockedis in the GPTBot group | 48,210 | 31,400 |
Claude-SearchBot | AI search | allowed | 0 | 0 |
PerplexityBot | AI search | allowed | 0 | 0 |
ChatGPT-User | Fetch on user request | allowedaccording to OpenAI, does not always follow the robots.txt | 0 | 0 |
GPTBot | AI training | blockedintended | 48,210 | 31,400 |
Demo project. The clicks from Search Console show how important the blocked pages are.
More about robots.txt monitoringsearchVIU saves every version and compares it line by line.
searchVIU compares every new robots.txt line by line with the previous one, per user agent. If a change blocks a search bot from Google or an AI search, the alert is critical. For training bots there is a warning, and for fetches on user request a notice with the severity Info. Each check run sends at most one email per project, and known changes are not emailed again.
Go to the alert list+ User-agent: OAI-SearchBotUser-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /basket/
You test a new robots.txt before rolling it out.
For teams with many hosts and frequent deployments.
The logs show whether an allowed bot really visits.
Your server logs show whether ChatGPT-User or ClaudeBot really fetch your pages, how often and which ones. They also reveal when the firewall blocks bots that the robots.txt allows.
What is left without JavaScript?
A bot only sees content that JavaScript adds to the page if it renders the page. searchVIU crawls every page in a real Chrome and compares the source code with the rendered page. That way you see what a bot finds without rendering.
| On the page | Source code | Rendered |
|---|---|---|
| Products | 0 | 36 |
| Internal links | 18 | 142 |
| Body text | 25 words | 410 words |
The right bots are allowed in.
- AI search allowedOAI-SearchBot, Claude-SearchBot, PerplexityBot
- Training under controlGPTBot blocked in its own group
- robots.txt unchanged96 checks since yesterday
Frequently asked questions about AI crawler monitoring
Does searchVIU have its own AI crawler?
No. AI crawler monitoring means that searchVIU monitors which AI bots may fetch your website. The searchVIU crawler is called searchVIUbot and crawls your website for SEO Monitoring.
Which bots does searchVIU recognize?
searchVIU knows the common bots for search, AI search, AI training, fetches on user request and SEO tools. These include, for example, Googlebot, GPTBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, ChatGPT-User, Claude-User and MistralAI-User. According to their operators, ChatGPT-User, Perplexity-User and meta-externalfetcher do not always follow the robots.txt. searchVIU therefore reports a block for these bots as a notice. According to Anthropic, Claude-User can be blocked via robots.txt. Go to the AI bot list
We have many domains and subdomains. Does that work?
Yes. searchVIU checks the robots.txt of every host from your live crawls, including country sites and old domains after a migration.
How quickly does the notification arrive?
searchVIU checks every 15 minutes. If a check run finds a change, you get at most one notification per project, like all alerts by email, Slack, Teams or webhook. searchVIU does not report known changes again.
Is this included in the price?
Yes. AI crawler monitoring is part of SEO Monitoring, from €399 per month with 50,000 URLs included. You can find all prices on the pricing page. Go to pricing
How is this different from AI log file analysis?
AI crawler monitoring shows which bots may fetch your pages. AI log file analysis shows which bots actually visit and how often. It is an add-on to SEO Monitoring. Go to AI log file analysis
Let's talk about your website.
In the demo, Michael shows you how searchVIU works. We use your website and a demo project for this.
- Request in three short steps
- Quick alignment on crawler access, NDA and Search Console
- Demo with your website and the demo project
- Start planned and supported together
Which website is it about?
Please enter your website and your business email.We need your business email for the demo. Without a company address, you can use our free tools instead.
What do you need?
Select as many as you like.
Who are you?
We only use your details for the demo. More in our privacy policy.
We need your business email for the demo.Please check your email address.Please enter your website.Please check your website address.The spam check could not be loaded. Please email us at info@searchviu.com.That didn't work. Please email us at info@searchviu.com.
Thanks! Now pick a time.
The calendar shows the open slots for your call with Michael.