How to identify and control searchVIUbot

What searchVIUbot does

searchVIU is a platform for SEO monitoring and crawling. Companies use it to analyze their own websites. They see which pages exist, how they are linked, what has changed since the last crawl and how a search engine sees the pages. searchVIUbot collects the data for this.

Most requests go to websites whose operators commissioned the analysis themselves. A small share goes to other websites, for example when an analysis fetches a linked page.

Without the operator's login details, searchVIUbot only fetches publicly accessible pages. It does not log in anywhere and does not submit forms.

How to identify searchVIUbot

searchVIUbot identifies itself in the user agent header. The product token is searchVIUbot/2.0, followed by the address of this page. There is a desktop and a mobile variant.

Desktop

Mobile

searchVIUbot renders every page in a real Google Chrome browser. The Chrome version in the string is the one actually running. It increases as soon as the browser is updated. So only check for the token searchVIUbot, never for the full string.

Different identifier per project

searchVIUbot is the default identifier. At the website operator's request, the identifier can be changed per project. This is needed, for example, when an operator wants to see on their own website exactly what Googlebot receives. That analysis then sends the Googlebot user agent or a custom string defined by the operator.

The user agent alone is therefore not reliable proof. Verification depends on the IP address, see IP address and verification.

Control via robots.txt

searchVIUbot reads your robots.txt before crawling and follows it. There are two cases.

Your robots.txt has a group for searchVIUbot

Then only these rules apply. They do not affect Googlebot or any other crawler.

Your robots.txt does not mention searchVIUbot

Then searchVIUbot follows your rules for Googlebot. The analysis is meant to show what a search engine can reach. searchVIUbot therefore does not fetch a page you block for Googlebot either.

If there is no group for Googlebot either, the rules for all crawlers apply, i.e. the group User-agent: *.

So if you want to allow Googlebot and exclude searchVIUbot, an entry for Googlebot is not enough. You need a separate group for searchVIUbot as shown above.

What searchVIUbot does not evaluate

searchVIUbot currently does not evaluate the Crawl-delay directive. If the crawl rate is too high for your servers, write to us. We will then throttle the crawl for your project.

searchVIUbot caches your robots.txt for up to 30 minutes. A change can therefore take up to 30 minutes to take effect.

IP address and verification

Anyone can copy a user agent. You can only be sure once you check where the request comes from. The searchVIU crawlers run on Google Cloud in the EU. New projects crawl from this fixed address.

If you maintain an allowlist, agree the release with us beforehand. We confirm in writing which address is set for your project and let you know if it changes.

If you want to check a single request, send us the date, the time with time zone and the IP address. We will tell you whether the request came from us. If something claims to be searchVIUbot and comes from an address we do not confirm, you can block it.

Crawl rate and care for your server

Every page is rendered in Chrome. This usually takes 6 to 15 seconds. A single thread therefore only fetches around 2 to 4 pages per minute. We set the crawl rate and the number of concurrent requests per project.

If a crawl should stop, an email to info@searchviu.com is enough. We cancel running crawls at any time. Throttling is usually quicker to arrange than a block, and the analysis your company commissioned keeps running.

Rendering and what is fetched

searchVIUbot opens every URL in Google Chrome (stable channel) and executes JavaScript. The browser loads CSS, JavaScript, fonts and images just like a normal visit. A page counts as finished when it has fully loaded and the network is idle. After 30 seconds, rendering is aborted.

• Over 300 known tracking, analytics and advertising addresses are blocked before the request goes out. This keeps your analytics figures accurate.
• searchVIU caches CSS, JavaScript and fonts jointly for all render instances. Your bundles are therefore not reloaded for every page. The HTML of every URL is always fetched fresh.
• searchVIUbot requests redirects one step at a time, for a maximum of 10 steps.
• searchVIUbot downloads PDF and Office files of up to 30 MB and extracts the text.

Allow or block searchVIUbot

Allow

Many firewalls and bot protection services do not let unknown crawlers through. If your company has commissioned searchVIU, allow the confirmed IP address. The safest option is a rule that checks the IP address and the token searchVIUbot together. In Cloudflare, for example, you can do this with a custom WAF rule that routes such requests past the bot protection.

Block

The simplest way is a separate group in your robots.txt, see Control via robots.txt. Blocking by IP address takes effect immediately, but it also prevents searchVIUbot from reading your robots.txt. If you only want to lower the crawl rate, write to us. We will then adjust the crawl for your project.

Contact

Questions, complaints, allowlisting or requests for a lower crawl rate go to info@searchviu.com. It helps if you include your domain and, for a specific request, the date, the time with time zone and the IP address.

How searchVIU handles data is described on the Security & GDPR page.