SEO crawler Every page rendered in a browser, every crawl kept for comparison

searchVIU crawls your whole website every week and renders every page with JavaScript. Earlier crawls are kept, so you see what has changed.

Request a demo
Millions of URLsper project, in the cloudCurrent Chromeevery page renderedsearchVIUbotwith a fixed IP6 months of historywith a screenshot per crawl
URL Historyexample-shop.com/tents/2-person/
Week 41Title changed in the rendered HTML
Diff
Week 40no change · screenshot saved
Ok
Week 3914 new internal links
Diff
Week 38no change · screenshot saved
Ok
Desktop crawlers

A crawl on your laptop is out of date after a week.

On large websites, desktop crawlers run out of memory and a crawl takes hours. In the end it shows the state of that one day.

Limits of your machine

Hundreds of thousands of URLs need memory and time. While the crawl runs, your computer mostly works for the crawler.

A snapshot without history

A single crawl shows the website as it is today. If a title was different last week or a page has gone missing since, you have to piece it together from old exports. That is also true when the crawler runs on a server.

Screaming Frog and searchVIU compared
Rendering

searchVIU opens every URL in a real browser.

searchVIU loads every page in a current Google Chrome and runs the JavaScript. It stores the raw and the rendered HTML, a screenshot, the network log and the HTTP headers. For every URL you see what your server delivers and what ends up in the browser.

More on JavaScript SEO
/tents/2-person/Source → rendered
<title>Tents for 2 people | Example Shop</title><title>Example Shop</title><link rel="canonical" href="https://example-shop.com/tents/2-person/">
Status code 200Canonical unchangedScreenshot saved
URL sources

With more sources than links alone, orphan pages show up too.

The crawler follows your internal links and adds further sources. That way you see pages that are not linked anywhere or have gone missing since the last crawl.

Internal links

The crawl starts on your website and follows every link.

XML sitemaps

URLs from your sitemaps are added, even if they are not linked internally.

Search Console

URLs with impressions on Google are crawled as well.

External links and earlier crawls

Externally linked URLs and all URLs from the last crawl close the gaps.

What searchVIU checks on every URL
Crawl history

Every crawl is kept, so every change shows.

searchVIU compares every crawl with the previous one. The topic reports show what has been added or dropped since the last crawl, from canonicals to structured data. For every single URL you can go back week by week, with screenshot and diff.

  • 6 months of historyby default, adjustable
  • Every URLwith screenshot and HTML diff per crawl
More on SEO Change Monitoring
Since the last crawlexample-shop.com · week 41
  • Status codes312 URLs now return 404
    new
  • Canonicals37 pages with a new canonical target
    new
  • Internal linking1,284 additional URLs linked
    changed
  • Structured dataunchanged
    same
In the cloud

searchVIU crawls large websites in the cloud.

The crawl runs in Google Cloud in the EU. Your computer stays free, even with millions of URLs.

Millionsof URLs per project
Every weekfull crawl, every URL rendered
Over 100checks per URL
6 monthsof history by default
For your IT team

The crawler goes easy on your server.

You can set the crawler's speed for each website. If your server responds with 429 or 5xx, the crawler lowers its rate automatically. For allowlisting, your IT team gets a fixed IP and the user agent searchVIUbot.

    Bot documentation for your IT team
    What the crawler respectsevery project
    • Crawl rate per websiteset to match your server and hosting
    • Half rate on 429Retry-After is respected
    • Slower on 5xx, automaticallythen the crawler speeds up again
    • Fixed IP and searchVIUbotfor allowlisting in your firewall and bot protection
    • 322 tracking endpoints blockedyour analytics data stays clean
    Settings

    We set up the crawl for your website.

    Every website is built differently. That is why we set up the crawl for your project and adjust it when the website changes.

    1Page types

    Analyze product, category or guide pages separately.

    2Custom extraction

    Extract additional fields from the HTML that matter for your website.

    3DEV and staging crawls

    Also password-protected, via Basic Auth, a custom header or IP allowlisting.

    4Quick Check

    Check a list of URLs on demand, outside the weekly crawl.

    For content behind consent banners or only reachable after a click, the crawler uses a custom script for your website.

    More on Staging Monitoring
    After the crawl

    The crawl is done and found nothing new.

    Most weeks are quiet. The crawl records that too, so you do not have to check yourself.

    • All URLs crawledincluding those from sitemaps and Search Console
    • No new error pagescompared with the last crawl
    • Server not throttledno 429 or 5xx during the crawl
    Week 40 · crawl without new errors

    Frequently asked questions about the crawler

    Does the crawler put load on our server?

    We set the speed for your website. If the server responds with 429, the crawler halves its rate and follows Retry-After. On 5xx it also slows down automatically.

    How do we allowlist the crawler?

    Via the fixed searchVIU IP or the user agent searchVIUbot. We agree the details for your IT team with you before the first crawl.

    Does searchVIUbot follow the robots.txt?

    Yes, by default. For test environments with “Disallow: /” we can set up a separate robots.txt that only applies to your crawls. Which group searchVIUbot reads is explained in the bot documentation.

    Does searchVIU render as mobile or desktop?

    By default, searchVIU crawls as a smartphone, because Google also crawls mobile first. If you need it, we switch your project to desktop or to both.

    How long does a crawl take?

    That depends on how fast we are allowed to crawl. Technically, searchVIU can crawl 100,000 URLs in a few hours. We agree the speed with you and your IT team beforehand. The full crawl runs every week, and Page Watch also checks your most important pages continuously, throughout the day.

    Does the crawl distort our analytics data?

    No. When rendering, searchVIU blocks 322 known tracking, analytics and advertising endpoints.

    We already use Screaming Frog. Why searchVIU?

    Screaming Frog is strong for analyses on your own computer. searchVIU crawls in the cloud every week without you having to do anything, and keeps every crawl for comparison. Your whole team sees the results in the same project.

    Where is our data stored?

    Your project data is stored in Google Cloud in the EU. searchVIU is certified to ISO/IEC 27001:2022. More on security

    What does the crawler cost?

    The crawler is part of SEO Monitoring, from €399 per month with 50,000 URLs included. Data consulting is included. You can find all prices on the pricing page.

    Demo

    Let's talk about your website.

    In the demo, Michael shows you how searchVIU works. We use your website and a demo project.

    1. Request in three short steps
    2. Quick alignment on crawler access, NDA and Search Console
    3. Demo with your website and the demo project
    4. Start planned and supported together
    1. Website
    2. Needs
    3. Contact

    Which website is it about?