robots.txt Monitoring If someone changes the robots.txt, searchVIU reports it.

searchVIU checks the robots.txt on all your hosts every 15 minutes. If a change locks out Google or AI bots, you get an alert.

Request a demo
Every 15 minuteson all hostsSearch and AI botswith the Google clicks of blocked URLsEvery versionwith a line-by-line diff
robots.txt Monitoringexample-shop.com · 6 minutes ago
robots.txt content changed2 lines added
Warning
GPTBot newly blockedAI training
Warning
OAI-SearchBot newly blockedAI search · 2,140 URLs with Google clicks
Critical
Versions and diff

Every version saved and every line compared

searchVIU saves every version of your robots.txt. When it changes, you see the diff per line and per user agent. The history shows when each version was live. searchVIU reports a state once, when it changes.

More about SEO alerts
www.example-shop.com/robots.txtChange found
User-agent: *
Disallow: /checkout/
- Disallow: /search/+ Disallow: /
Website blocked for all botsAlert sent 6 minutes after the change
Bot matrix

Which bot is allowed in, and how many clicks depend on a block?

The bot matrix sorts known bots into classes: search engines, AI search, AI training, fetches on user request and SEO tools. For each bot you see whether it is allowed, which URLs are blocked and how many Google clicks those URLs get.

    More about AI crawler monitoring
    BotClassStatusGoogle clicks on these URLs
    GooglebotSearch engineallowed0
    OAI-SearchBotAI searchblocked2,140
    PerplexityBotAI searchallowed0
    GPTBotAI trainingblocked2,140
    AhrefsBotSEO toolpartly310
    example-shop.com · clicks in the last 28 days2 of 5 blocked
    AI bots

    searchVIU shows which AI bot a rule hits.

    GPTBot collects training data, while OAI-SearchBot fetches pages for answers in ChatGPT. A rule that is too broad hits both. searchVIU shows you which AI search bots a change affects.

    Go to AI visibility
    All hosts

    searchVIU checks every host, including subdomains and country domains.

    The basis is the hosts in your crawl profiles. searchVIU checks each of these robots.txt files at the same 15-minute interval.

      More about SEO Monitoring
      Hostsexample-shop · today
      www.example-shop.comunchanged for 9 weeks
      Ok
      www.example-shop.co.ukunchanged for 9 weeks
      Ok
      www.example-shop.iereachable again · outage of about 2 hours
      High
      shop.example-shop.comSubdomain · unchanged
      Ok
      What gets reported

      From a complete outage to a missing sitemap reference

      searchVIU classifies the events on your robots.txt by impact. Blocking the whole website is critical, while a bot that is allowed again is only an info.

      Website completely blocked“Disallow: /” for all bots, for example when the staging robots.txt goes live
      Bot blocked or allowed againwith affected URLs and their Google clicks
      Unreachable or server errorreported as soon as the outage is confirmed. When the file is back, the duration of the outage is added.
      HTML instead of robots.txtfor example a redirect to the homepage after a relaunch
      Sitemap reference removedor newly added
      Content signals changedfor search, AI answers or AI training
      All robots.txt events in the alert list
      robots.txt today

      96 checks today without a single change

      Most of the time, the robots.txt stays as it is. searchVIU still records every check.

      • All hosts reachablechecked every 15 minutes
      • No change to bot rulesversion history unchanged
      • Sitemap references completeas in the last version
      Today · unchanged 96 times

      Frequently asked questions

      How often does searchVIU check the robots.txt?
      Every 15 minutes, on every host in your crawl profiles. searchVIU reports a state once, when it changes. Each check run produces at most one email per project.
      Which bots does searchVIU know?
      The bot matrix sorts known bots into classes: search engines, AI search, AI training, fetches on user request and SEO tools. These include, for example, Googlebot, GPTBot and OAI-SearchBot. Bots such as ChatGPT-User fetch pages at a user's request and, according to their operators, do not reliably follow the robots.txt. searchVIU therefore reports blocking them only as an info.
      Does this also apply to subdomains and country domains?
      Yes, to every host listed in your crawl profiles.
      Can we test a planned change first?
      Yes. In test mode you run through a new version of the robots.txt without anything being saved or emailed.
      How much does robots.txt monitoring cost?
      robots.txt monitoring is included in SEO Monitoring, from €399 per month with 50,000 URLs included. You can find all prices on the pricing page.
      Demo

      Let's talk about your website.

      In the demo, Michael shows you how searchVIU works. We use your website and a demo project for this.

      1. Request in three short steps
      2. Quick alignment on crawler access, NDA and Search Console
      3. Demo with your website and the demo project
      4. Start planned and supported together
      1. Website
      2. Needs
      3. Contact

      Which website is it about?