Enterprise SEO crawling The SEO crawler for websites with millions of URLs
searchVIU crawls your entire website in the cloud and renders every single page in Chrome. Every week, you get the complete picture, page by page.
Example values from an active project, as of November 2026.
What does it take to crawl a website with millions of URLs?
You need a crawler in the cloud that renders every page and sticks to the crawl rate your IT sets. searchVIU does exactly that.
- Millions of URLs in one crawlsearchVIU crawls websites with millions of URLs in Google Cloud in the EU. Several million URLs a week are no problem.
- Every page rendered in ChromeEvery single page runs through Chrome. You see what Google sees once JavaScript has run.
- Your IT sets the crawl rateThe number of concurrent requests is set per project and agreed with your IT. searchVIU renders around 2 to 3 pages per minute per concurrent request.
Is a desktop crawler enough for a large website?
For individual analyses, it often is. Screaming Frog writes in its FAQ that most modern machines can handle crawls of millions of URLs. It is a different story when the whole website has to be crawled with JavaScript every week.
- SizeAccording to the vendor, the SEO Spider is recommended for crawls below 2 million URLs and can crawl far more with enough memory and storage.
- DurationFor 1 million pages at 5 URLs per second, Screaming Frog gives over two days. The computer has to keep running all that time.
- JavaScriptBy default, the SEO Spider crawls the raw HTML. The vendor recommends using JavaScript crawling selectively because it is slower and more resource-intensive.
Information from Screaming Frog's FAQ and SEO Spider guides, retrieved November 13, 2026.
Screaming Frog and alternatives comparedHow do you crawl a large website without overloading the server?
You limit how many requests reach the server at the same time, agree this value with IT and let the crawler slow down as soon as the server asks for it. The eight steps apply to any crawler. Each one shows how searchVIU handles it.
Before the first crawl, clarify how many concurrent requests the server can handle and at what times. At searchVIU, this alignment is part of every project kickoff.
Firewalls and CDNs often block unknown bots that send many requests. searchVIU crawls as searchVIUbot from the fixed IP 35.190.196.200, which your IT allowlists in advance.
Block what should not be crawled. searchVIUbot follows its own group, otherwise the rules for Googlebot. It does not read Crawl-delay. The crawl rate is set per project instead.
The number of concurrent requests determines the load on the server. In searchVIU, it is set per project, and on request with a fixed cap on requests per minute.
Run most of the crawl at times with few visitors. searchVIU sets up time windows for this, for example a tenth of the requests during the day and full speed at night.
With HTTP 429, a server signals too many requests. The entire searchVIU crawl then halves its rate, and again with every further 429. After a minute without a 429, it gradually speeds up.
The Retry-After header tells a crawler how long to wait. searchVIUbot pauses for that long, at most 5 minutes at a time, and retries a URL at most 3 times.
For a maintenance window or a traffic peak, you need to be able to stop a crawl. In searchVIU, a running crawl can be paused and resumed later.
How long does it take to crawl a large website?
That depends on how many concurrent requests your server can handle and how long the pages take to render. searchVIU renders around 2 to 3 pages per minute per concurrent request and has not yet hit a limit of its own. Usually, it is the website itself that makes a slower crawl the sensible choice.
| Example | Pages per minute | 100,000 URLs | 1 million URLs |
|---|---|---|---|
| 15 concurrent requests | around 40 | around 1.7 days | – |
| 70 concurrent requests | around 135 | around 12 hours | around 5 days |
| Project with a high crawl rate | around 290 | around 6 hours | around 2.5 days |
Example values from active projects, as of November 2026. They are based on experience and are no guarantee.
What is ready after the crawl?
When the crawl is finished, you have every URL rendered, checked and sorted by page type. Crawling a website for technical SEO analysis
To the SEO Audit
To the list of checks
To SEO Change Monitoring
To the SEO Crawler
To the SEO Data Warehouse
To log file analysis
Where is the crawl data stored?
Your project data is stored in the EU. searchVIU is certified to ISO/IEC 27001:2022 and has a SOC 2 Type 2 report. SEO crawlers with data in the EU compared
The crawl completes every week, and your IT knows what reaches the server.
- Full crawl completeevery URL rendered in Chrome
- Crawl rate as agreedno 429 responses
- No new findings in the crawlcompared with the previous week
Frequently asked questions about crawling large websites
Which SEO crawler is right for enterprise websites?
For websites with millions of URLs, cloud crawlers are the main option. Botify, Lumar, Oncrawl and JetOctopus mention crawls of millions of URLs on their websites and target enterprise teams. Sitebulb Cloud states 10 million URLs per audit as its limit. Screaming Frog recommends its desktop crawler for crawls below 2 million URLs.
searchVIU crawls large and complex websites in the cloud and renders every single page in Chrome. The full crawl runs every week, your project data stays in the EU, and SEO Monitoring costs from €399 per month with 50,000 URLs included. For an overview, see the comparison of enterprise SEO crawlers.
Can searchVIU crawl websites with millions of URLs?
Yes. searchVIU crawls websites with millions of URLs and renders every single page in Chrome. Several million URLs a week are no problem, and there is no upper limit.
Does searchVIU render every URL with JavaScript on large websites?
Yes, searchVIU renders every single page in Google Chrome, even with millions of URLs. For each URL, you see the raw and the rendered HTML and can tell what JavaScript changes in the title, canonical, content and links. You can read more on the JavaScript SEO page. More on checking JavaScript rendering
Can an SEO crawler overload my server?
Yes, if it sends too many requests at the same time. Screaming Frog writes in its documentation that in very extreme cases a crawl can overload a server and crash it. More commonly today, a security platform blocks the crawler.
That is why searchVIU limits concurrent requests per project and agrees the value with your IT. If the server responds with HTTP 429, the crawl halves its rate. On a 5xx, searchVIUbot waits 10 seconds before fetching the URL again. How your IT allowlists the crawler is covered in the help article on searchVIUbot in your firewall, WAF and CDN.
How long does a crawl of 1 million URLs take?
Based on values from active projects, it takes around 2.5 to 5 days. 70 concurrent requests add up to around 135 pages per minute, which means around 5 days for 1 million URLs. A project with a higher crawl rate runs at around 290 pages per minute and crawls 1 million URLs in around 2.5 days.
These are example values from November 2026 and no guarantee. How fast your crawl runs depends on how many concurrent requests your server can handle and how long the pages take to render.
How much does it cost to crawl a large website?
SEO Monitoring costs from €399 per month with 50,000 URLs included. Rendering is included in SEO Monitoring. For larger websites, the calculator on the pricing page shows guide prices for up to 1 million URLs. Above that, you get a quote. You can find all tiers on the pricing page.
Let's talk about your website.
In the demo, Michael shows you how searchVIU works, using your website and a demo project.
- Request in three short steps
- Quick alignment on crawler access, NDA and Search Console
- Demo with your website and the demo project
- Start planned and supported together
Which website is it about?
Please enter your website and your business email.We need your business email for the demo. Without a company address, you can use our free tools instead.
What do you need?
Select as many as you like.
Who are you?
We only use your details for the demo. More in our privacy policy.
We can't send the request without the spam protection checkbox.
We need your business email for the demo.Please check your email address.Please enter your website.Please check your website address.The spam check could not be loaded. Please email us at info@searchviu.com.That didn't work. Please email us at info@searchviu.com.

Thanks! Now pick a time.
The calendar shows the open slots for your call with Michael.