Enterprise SEO crawling The SEO crawler for websites with millions of URLs

searchVIU crawls your entire website in the cloud and renders every single page in Chrome. Every week, you get the complete picture, page by page.

Every URL rendered in Chromeeven with millions of URLsFull crawlevery week, with Page Watch in betweenFixed IP and its own user agentso your IT can allowlist the crawler
Crawl profilebeispiel-shop.de · full crawl
User agentown rules in robots.txt possiblesearchVIUbot
IP addressfixed, for firewall and CDN35.190.196.200
Concurrent requestsagreed with IT70
Pages per minuteeach rendered in Chromearound 135
On HTTP 429Retry-After is respectedhalve the rate

Example values from an active project, as of November 2026.

In short

What does it take to crawl a website with millions of URLs?

You need a crawler in the cloud that renders every page and sticks to the crawl rate your IT sets. searchVIU does exactly that.

    • Millions of URLs in one crawlsearchVIU crawls websites with millions of URLs in Google Cloud in the EU. Several million URLs a week are no problem.
    • Every page rendered in ChromeEvery single page runs through Chrome. You see what Google sees once JavaScript has run.
    • Your IT sets the crawl rateThe number of concurrent requests is set per project and agreed with your IT. searchVIU renders around 2 to 3 pages per minute per concurrent request.
    Cloud or desktop

    Is a desktop crawler enough for a large website?

    For individual analyses, it often is. Screaming Frog writes in its FAQ that most modern machines can handle crawls of millions of URLs. It is a different story when the whole website has to be crawled with JavaScript every week.

    • SizeAccording to the vendor, the SEO Spider is recommended for crawls below 2 million URLs and can crawl far more with enough memory and storage.
    • DurationFor 1 million pages at 5 URLs per second, Screaming Frog gives over two days. The computer has to keep running all that time.
    • JavaScriptBy default, the SEO Spider crawls the raw HTML. The vendor recommends using JavaScript crawling selectively because it is slower and more resource-intensive.

    Information from Screaming Frog's FAQ and SEO Spider guides, retrieved November 13, 2026.

    Screaming Frog and alternatives compared
    Step by step

    How do you crawl a large website without overloading the server?

    You limit how many requests reach the server at the same time, agree this value with IT and let the crawler slow down as soon as the server asks for it. The eight steps apply to any crawler. Each one shows how searchVIU handles it.

    1Agree the crawl rate with IT

    Before the first crawl, clarify how many concurrent requests the server can handle and at what times. At searchVIU, this alignment is part of every project kickoff.

    2Make the crawler identifiable

    Firewalls and CDNs often block unknown bots that send many requests. searchVIU crawls as searchVIUbot from the fixed IP 35.190.196.200, which your IT allowlists in advance.

    3Check robots.txt

    Block what should not be crawled. searchVIUbot follows its own group, otherwise the rules for Googlebot. It does not read Crawl-delay. The crawl rate is set per project instead.

    4Limit concurrent requests

    The number of concurrent requests determines the load on the server. In searchVIU, it is set per project, and on request with a fixed cap on requests per minute.

    5Stagger the crawl rate by time of day

    Run most of the crawl at times with few visitors. searchVIU sets up time windows for this, for example a tenth of the requests during the day and full speed at night.

    6Slow down on 429

    With HTTP 429, a server signals too many requests. The entire searchVIU crawl then halves its rate, and again with every further 429. After a minute without a 429, it gradually speeds up.

    7Respect Retry-After

    The Retry-After header tells a crawler how long to wait. searchVIUbot pauses for that long, at most 5 minutes at a time, and retries a URL at most 3 times.

    8Be able to pause the crawl

    For a maintenance window or a traffic peak, you need to be able to stop a crawl. In searchVIU, a running crawl can be paused and resumed later.

    Read the searchVIUbot documentation
    Worked example

    How long does it take to crawl a large website?

    That depends on how many concurrent requests your server can handle and how long the pages take to render. searchVIU renders around 2 to 3 pages per minute per concurrent request and has not yet hit a limit of its own. Usually, it is the website itself that makes a slower crawl the sensible choice.

    ExamplePages per minute100,000 URLs1 million URLs
    15 concurrent requestsaround 40around 1.7 days–
    70 concurrent requestsaround 135around 12 hoursaround 5 days
    Project with a high crawl ratearound 290around 6 hoursaround 2.5 days

    Example values from active projects, as of November 2026. They are based on experience and are no guarantee.

    After the crawl

    What is ready after the crawl?

    When the crawl is finished, you have every URL rendered, checked and sorted by page type. Crawling a website for technical SEO analysis

    Analysis by page typeThe “SEO Issues” cockpit shows you the findings as a worklist by page type.
    To the SEO Audit
    Over 100 checksEvery URL goes through over 100 checks, sorted by affected pages and clicks.
    To the list of checks
    Weekly crawl plus Page WatchThe full crawl runs every week. In between, Page Watch checks your most important pages continuously, throughout the day.
    To SEO Change Monitoring
    Crawl historyEvery crawl is stored, by default with the HTML and a screenshot of each URL for six months. That way, you can compare every week with the one before.
    To the SEO Crawler
    SEO Data WarehouseThe crawl data sits in BigQuery in the EU, linked per URL with Search Console and GA4. Direct access to BigQuery is available on request.
    To the SEO Data Warehouse
    Log file analysisThe add-on shows which URLs Googlebot and AI bots actually request and where crawl budget gets lost.
    To log file analysis
    Security

    Where is the crawl data stored?

    Your project data is stored in the EU. searchVIU is certified to ISO/IEC 27001:2022 and has a SOC 2 Type 2 report. SEO crawlers with data in the EU compared

    More about security
    Every week

    The crawl completes every week, and your IT knows what reaches the server.

    • Full crawl completeevery URL rendered in Chrome
    • Crawl rate as agreedno 429 responses
    • No new findings in the crawlcompared with the previous week
    Week 46 · all quiet

    Frequently asked questions about crawling large websites

    Which SEO crawler is right for enterprise websites?

    For websites with millions of URLs, cloud crawlers are the main option. Botify, Lumar, Oncrawl and JetOctopus mention crawls of millions of URLs on their websites and target enterprise teams. Sitebulb Cloud states 10 million URLs per audit as its limit. Screaming Frog recommends its desktop crawler for crawls below 2 million URLs.

    searchVIU crawls large and complex websites in the cloud and renders every single page in Chrome. The full crawl runs every week, your project data stays in the EU, and SEO Monitoring costs from €399 per month with 50,000 URLs included. For an overview, see the comparison of enterprise SEO crawlers.

    Can searchVIU crawl websites with millions of URLs?

    Yes. searchVIU crawls websites with millions of URLs and renders every single page in Chrome. Several million URLs a week are no problem, and there is no upper limit.

    Does searchVIU render every URL with JavaScript on large websites?

    Yes, searchVIU renders every single page in Google Chrome, even with millions of URLs. For each URL, you see the raw and the rendered HTML and can tell what JavaScript changes in the title, canonical, content and links. You can read more on the JavaScript SEO page. More on checking JavaScript rendering

    Can an SEO crawler overload my server?

    Yes, if it sends too many requests at the same time. Screaming Frog writes in its documentation that in very extreme cases a crawl can overload a server and crash it. More commonly today, a security platform blocks the crawler.

    That is why searchVIU limits concurrent requests per project and agrees the value with your IT. If the server responds with HTTP 429, the crawl halves its rate. On a 5xx, searchVIUbot waits 10 seconds before fetching the URL again. How your IT allowlists the crawler is covered in the help article on searchVIUbot in your firewall, WAF and CDN.

    How long does a crawl of 1 million URLs take?

    Based on values from active projects, it takes around 2.5 to 5 days. 70 concurrent requests add up to around 135 pages per minute, which means around 5 days for 1 million URLs. A project with a higher crawl rate runs at around 290 pages per minute and crawls 1 million URLs in around 2.5 days.

    These are example values from November 2026 and no guarantee. How fast your crawl runs depends on how many concurrent requests your server can handle and how long the pages take to render.

    How much does it cost to crawl a large website?

    SEO Monitoring costs from €399 per month with 50,000 URLs included. Rendering is included in SEO Monitoring. For larger websites, the calculator on the pricing page shows guide prices for up to 1 million URLs. Above that, you get a quote. You can find all tiers on the pricing page.

    Demo

    Let's talk about your website.

    In the demo, Michael shows you how searchVIU works, using your website and a demo project.

    1. Request in three short steps
    2. Quick alignment on crawler access, NDA and Search Console
    3. Demo with your website and the demo project
    4. Start planned and supported together
    1. Website
    2. Needs
    3. Contact

    Which website is it about?