AI bots in robots.txt When AI bots come knocking, searchVIU shows who gets in.

searchVIU checks your robots.txt every 15 minutes for every bot. You can see which AI bots are blocked and how many Google clicks the affected pages receive.

Request a demo
every 15 minutesrobots.txt checkedBot matrixsorted by purposeLog file analysiswhich bots really visitAI Visibilityweekly in ChatGPT, Gemini and AI Mode
Alertsbeispiel-shop.de · today
OAI-SearchBot blockedrobots.txt changed at 09:45 · 48,210 URLs with 31,400 Google clicks
Critical
New group for Claude-Userrobots.txt changed at 09:45 · Disallow: /konto/
Warning
GPTBot still blockedAI training · own group, unchanged
Info
Many bots

One provider sends several bots, and a rule can easily hit the wrong one

OpenAI sends GPTBot for training and OAI-SearchBot for search in ChatGPT. If OAI-SearchBot is in the same group as GPTBot, ChatGPT search is blocked as well.

AI trainingGPTBot and other training bots collect content for language models.
AI searchOAI-SearchBot, Claude-SearchBot or PerplexityBot fetch content for answers with sources.
User-initiated fetchesChatGPT-User, Claude-User or Perplexity-User fetch a page when someone asks for it in a chat. According to their operators, ChatGPT-User and Perplexity-User do not always follow robots.txt, while Claude-User does.
Bot matrix

For every bot, you can see how many Google clicks a block affects

The bot matrix assigns every bot to a class, from search to AI search, AI training, user-initiated fetches and SEO tools. For every bot, you can see whether it is allowed in, how many URLs are blocked and how many Google clicks these URLs receive.

    More about AI crawler monitoring
    BotAccessGoogle clicks on these URLsClass
    Googlebotallowed0Search
    OAI-SearchBotblockedall 48,210 URLs31,400AI search
    ChatGPT-Userallowedaccording to OpenAI, does not always follow robots.txt0User-initiated fetch
    Claude-Userpartial/konto/0User-initiated fetch
    GPTBotblockedall 48,210 URLs, intentional31,400AI training
    AhrefsBotallowed0SEO tools

    The clicks from Search Console show how important the blocked pages are. Sample data.

    Changes

    When a new line appears in your robots.txt, searchVIU finds it

    searchVIU checks your robots.txt every 15 minutes and stores every version. The diff shows what has changed per line and user agent. You can check planned changes in test mode beforehand.

    More about robots.txt monitoring
    www.beispiel-shop.de/robots.txtChange at 09:45
    User-agent: GPTBot
+ User-agent: OAI-SearchBotDisallow: /

User-agent: *
Disallow: /checkout/
    OAI-SearchBot blocked for all URLsAlert sent to the team
    Log file analysis

    robots.txt says who may enter, and the logs show who actually comes

    Log file analysis shows which AI bots really fetch your pages. This includes fetches for individual answers and requests for the llms.txt. searchVIU flags bots that merely pose as a known bot as fake bots.

    • Log sourcesCloudflare, upload, FTP or Amazon S3
    • Fake botsshown separately
    More about AI log file analysis
    Requests by AI botsbeispiel-shop.de · last 7 days
    ChatGPT-User38%
    OAI-SearchBot21%
    GPTBot17%
    PerplexityBot11%
    Claude-SearchBot8%
    Fake bots5%
    AI answers

    Does the access you allow show up in AI answers?

    AI Visibility Monitoring checks every week whether ChatGPT, Gemini and Google AI Mode mention you and which pages they cite as sources. This shows you whether your allowed pages then appear as sources.

    More about AI Visibility Monitoring

    Frequently asked questions about AI bots

    Should we block GPTBot?
    That is your decision. searchVIU shows what purpose a bot serves, how many URLs a block affects and how many Google clicks these URLs receive. With a separate group for GPTBot, you can block training while OAI-SearchBot can still fetch pages.
    Does a rule for GPTBot automatically block OAI-SearchBot too?
    Only if both are in the same group or a rule applies to all bots. The bot matrix shows for each bot individually whether it is blocked.
    Our logs are not on Cloudflare.
    searchVIU also reads logs via upload, via FTP or from Amazon S3. We can connect further sources. To log file analysis
    Is an llms.txt worth it?
    That depends on whether AI services fetch it, and that is exactly what log file analysis shows. Whether it affects mentions in AI answers is an open question. To llms.txt monitoring
    How accurate are the figures from AI answers?
    They are approximations, because AI answers vary from query to query. searchVIU measures every week with the same prompts, which makes the trend per engine visible.
    How much does it cost?
    robots.txt monitoring is part of SEO Monitoring, from €399 per month with 50,000 URLs included. Log file analysis and AI visibility are add-ons to SEO Monitoring, each from €199 per month. You can find all prices on the pricing page. To pricing
    Demo

    Let's talk about your website.

    In the demo, Michael shows you how searchVIU works, using your website and a demo project.

    1. Request in three short steps
    2. Quick alignment on crawler access, NDA and Search Console
    3. Demo with your website and the demo project
    4. Start planned and supported together
    1. Website
    2. Needs
    3. Contact

    Which website is it about?