The six crawlers by criteria
The tables show the crawlers in the same order as the cards. The information on other vendors comes from their websites, pricing pages and documentation, as of November 16, 2026. The most important sources are listed at the end of the page. “Not documented” means that the vendor's public pages said nothing about it.
Approach, size and JavaScript rendering
| Crawler | Approach | Scale according to the vendor | JavaScript rendering and what it costs |
|---|
| searchVIU (ours) | Cloud. Full crawl every week, and in between, Page Watch checks the most important pages. | Websites with millions of URLs, and there is no upper limit. Several million URLs a week are no problem. | Every URL is rendered in Chrome. Rendering is included in SEO Monitoring. |
| Lumar | Cloud. Scheduled crawls, up to hourly according to the FAQ. | Websites with “hundreds of thousands (or millions) of pages”. Lumar does not name an upper limit, and billing is based on a URL allowance per billing period. | Uses Google's rendering and parsing engine, according to the crawler page. Whether rendered pages use more allowance is not documented. |
| Botify | Cloud. Scheduled crawls, from twice a month to daily depending on the plan, according to the contract annex. | “Millions of pages in a single crawl” according to the product page. An announcement from 2017 cites 25 million pages per crawl, and the 2026 contract annex lists monthly allowances of 2 million to 50 million pages per plan. | The same rendering engine as Googlebot, according to the product page. According to the contract annex, JavaScript is an option that the account manager activates. What it costs is not documented. |
| Oncrawl | Cloud. Scheduled crawls, and you set the frequency for each crawl configuration. | “Millions of URLs without artificial limits” according to the crawler page. Billing is based on a monthly token quota. | Rendering “at any scale” according to the crawler page. According to the help center, a crawl with JavaScript uses three times as many crawl tokens. |
| JetOctopus | Cloud. Scheduled crawls, and logs come in independently of them. | “10M, 50M, 100M+ pages” according to the enterprise page. The public plans include 500,000 to 5 million URLs per month, and above that there is the Unlimited plan on request. | Rendering like Googlebot, according to the vendor. According to the FAQ, one rendered page counts as two crawled pages. |
| Audisto | Cloud. Scheduled crawls, up to hourly in the Monitoring plan and daily in the Crawler plan. | “20 million HTML pages or more” according to the crawler page, without a credit system. | Included in both plans. |
Crawl rate, log files and data
Crawl rates given by vendors are maximum values. In practice, the website's server usually limits how fast a crawl can run.
| Crawler | Crawl rate and control | Log files | Data and export |
|---|
| searchVIU (ours) | searchVIU sets the number of concurrent requests per project and coordinates it with your IT team. If the server responds with HTTP 429, the crawl halves its rate. Scheduling the rate by time of day and a fixed cap on requests per minute are possible. | Log file analysis for Googlebot and AI bots as an add-on, analyzed together with the crawl data. | Crawl, Search Console and GA4 in one project, over 80 standard reports and an MCP server. BigQuery access is available on request. |
| Lumar | Maximum crawl speed adjustable, with a higher speed during quiet periods, for example from 1am to 5am. The “Crawl Safeguard” pauses the crawl automatically when a failure rate threshold is reached. Lumar cites up to 450 URLs per second non-rendered and 350 rendered, test values from 2022. | Log data comes into the analysis via Logz.io, Splunk or upload. | API and integrations according to the pricing page. |
| Botify | Maximum crawl speed adjustable. According to the help center, Botify adjusts the crawl speed automatically based on the server's response time and crawls unverified websites at up to three pages per second. The crawler can crawl over 200 pages per second, according to the help center. | LogAnalyzer as a separate module. | Exports to AWS S3 and Google Cloud Storage, among others. |
| Oncrawl | Maximum crawl speed in URLs per second adjustable, also while the crawl is running. According to the help center, more than 1 URL per second requires a verified domain. The recommended speed is 2 to 5 URLs per second. | Log Analyzer as a separate module. | Combines crawl, log and performance data, according to the website. There is also an API. |
| JetOctopus | You control the crawl rate through the number of threads. According to the FAQ, up to 250 pages per second, and at 5 threads, 1 million URLs take one to two days. | Log Analyzer in every plan. | Crawl, logs, Search Console and GA4 in one product, all modules in every plan. |
| Audisto | Up to 300 URLs per second according to the crawler page, with “intelligent overload protection” for server problems. | not documented | API and exports in every plan, according to the pricing page. |
Analysis, alerts and custom check rules
With millions of URLs, the analysis decides whether a crawl turns into work your team can act on. The middle column shows when a tool sends alerts and what it checks between two full crawls.
| Crawler | Analysis | Monitoring and alerts | Custom check rules |
|---|
| searchVIU (ours) | Every crawl is analyzed by page type, with over 100 checks per URL and over 80 standard reports. The comparison with the previous crawl shows what is new and what has been fixed. The HTML and a screenshot of every URL are stored for six months. | Page Watch checks your most important pages continuously, throughout the day, plus the robots.txt and the sitemaps on their own schedule. Alerts are weighted by affected URLs and Google clicks and arrive by email, Slack, Teams or webhook. | searchVIU sets up custom alert rules for you, also per segment. Custom extraction works with XPath and regex, and searchVIU builds custom reports as needed. |
| Lumar | You create segments by page type, category or country yourself and choose them in every report. Trend lines show metrics across several crawls, in the report also per segment. | You set alerts on any report, any segment and any extraction, with your own or an automatically adjusted threshold. Every completed crawl is evaluated, and critical alerts arrive by email, Slack or Teams. Monitoring between crawls is not documented. | You create custom extractions with regex yourself in the project settings. On top of that come up to 25 custom metrics per project, with AI creating the code from your specification. |
| Botify | SiteCrawler shows the data by segment, and you can filter reports by segment. You can compare any two crawls, and according to the plan overview, trend reports go back 24 months. | AlertPanel checks selected URLs, the robots.txt and logs daily. Custom alerts on any crawl metric run at the end of each regular crawl. Alerts are triggered by statistically significant deviations and arrive by email, Slack or Teams. | At Botify, custom fields based on CSS selectors, XPath or regex are called HTML Extracts. For setup, the help center points to the project settings and the Customer Success Manager. |
| Oncrawl | You create segmentations with page groups yourself, based on any metric and also for past crawls. Crawl over Crawl compares two crawls, and the Scorecard tracks one metric over time. | Custom alerts on pages, links or the crawl comparison, with a threshold you set as a number or a percentage. Alerts are checked at the end of each analysis and sent by email, Slack or Teams. | You create custom fields with XPath, regex or both in the crawl settings, up to 100 rules per crawl configuration. Data scraping is an optional feature that not every plan includes. |
| JetOctopus | You create segments yourself with no limit on their number and use them for crawl, logs and Search Console. According to the vendor, Compare Crawls compares up to 10 crawls with trend lines. | According to the vendor, log and Search Console alerts run without a crawl, and crawl alerts run on the cadence of scheduled crawls. You set the condition and threshold yourself, and alerts arrive by email, Slack or Teams. | You create up to 50 extraction rules per crawl yourself, with CSS selectors or regex, and according to the vendor also with XPath and JavaScript since 2024. Alerts can be combined with extractions. |
| Audisto | Up to 70 custom segments per crawl, called clusters at Audisto, divide the website by page type, for example. According to the pricing page, the KPI history is unlimited, and 3 crawls are retained for in-depth analysis. | Alerts are tied to custom checks with an absolute or percentage threshold. Critical alerts are sent as soon as the threshold is reached, the others when the crawl is finished. The Monitoring plan schedules crawls up to hourly. The help center names email, and the product page also names Jira, Slack and Teams. | You write custom tests yourself, in the visual editor or as text, with regex, XPath or CSS selectors. The Monitoring plan includes 200 requirements per crawl, the Crawler plan none. |
Data security and price
The information on data location and certificates comes from the vendors' security and legal pages. Prices of other vendors are given as the vendor states them.
| Crawler | Headquarters and data location | Certificates | Price |
|---|
| searchVIU (ours) | Bonn. Project data in the EU. | ISO/IEC 27001:2022 and SOC 2 Type 2. | From €399 per month with 50,000 URLs included, excl. VAT. The calculator shows guide prices up to 1 million URLs, and above that you get a quote. |
| Lumar | London. Amazon Web Services, Lumar does not name a region. | SOC 2 Type 2. | Not public. |
| Botify | Paris. According to the DPA, data of customers based in the EU is processed only in the EU. | SOC 2 Type II. | Not public. |
| Oncrawl | Mérignac near Bordeaux, part of BrightEdge since 2022. Google Cloud, EU hosting as an option. | ISO 27001, SOC 2 Type I and Type II. | Not public. |
| JetOctopus | Tallinn. The 2019 privacy policy names OVH in France, the security page names no location. | not documented | Growth €299 or €399 per month depending on the billing option, with 500,000 URLs per month. Whether net or gross is not stated. |
| Audisto | Gernsbach. Data centers in the EU, Audisto does not name the location. | not documented | Crawler from €99, Monitoring from €199 per month excl. VAT, each for 10,000 pages per crawl. |
Cloud or desktop?
A desktop crawler runs as a program on your computer, a cloud crawler on the vendor's servers. For a single audit of up to several hundred thousand URLs, a desktop crawler such as Screaming Frog or Sitebulb is often enough. Screaming Frog describes the limits of its own tool openly, and these statements help with the decision.
- Size. According to the SEO Spider FAQ, the SEO Spider is “recommended for crawls below 2 million URLs” and can handle far more with the right hardware. For around 10 million URLs, the tutorial How To Crawl Large Websites names 16 GB of RAM.
- Duration. “A 1 million page website at an average crawl rate of 5 URLs per second will take over two days to crawl,” Screaming Frog writes in the same tutorial.
- Runtime. The computer has to keep running during the crawl. For the cloud, Screaming Frog describes a do-it-yourself setup on your own virtual machine and gives 3.1 million URLs in around two days as an example.
- JavaScript. By default, the SEO Spider crawls without rendering. Screaming Frog recommends using JavaScript crawling selectively, because it is slower and needs more resources.
- Partial crawls. For large websites, the tutorial often advises crawling only part of the site. It asks “So, why crawl 5m URLs, when 50k is enough?”
Screaming Frog itself sees this differently. According to the FAQ, large crawls do not need a server, because modern computers can handle crawls with millions of URLs alongside normal work. Sitebulb offers both options, with 500,000 URLs per audit in the Pro license on the desktop and up to 10 million URLs per audit in the cloud, according to the Sitebulb FAQs.
Both tools are built for audits. They do not offer ongoing monitoring between crawls. Screaming Frog sends an email after a scheduled crawl if you want, and Sitebulb recommends scheduled audits. That is why both appear here as a point of reference.
A cloud crawler pays off as soon as one of these points applies.
- The entire website should be crawled regularly, with every URL rendered.
- Your team wants to get alerts when important pages change.
- Several people work with the same crawl data and its history.
- Your IT team wants to approve the crawler by IP address and user agent.
- Crawl data should be linked with Search Console, analytics or log files.
A desktop crawler fits if you run audits yourself and as needed, the website suits the power of your computer, and the crawl data should stay with you. You can find more in the comparisons Screaming Frog alternatives and Sitebulb alternatives.
Which crawler fits which website size?
There are no fixed thresholds, because rendering, page types and the server all play a part. The vendors' figures serve as a guide.
| Website size | Suitable crawlers | Basis according to the vendor |
|---|
| up to around 500,000 URLs | For a single audit, Screaming Frog or Sitebulb Desktop are enough. As soon as you want to crawl regularly and get alerts, any of the six cloud crawlers fits. | Sitebulb names 500,000 URLs as the default limit of the Pro license. Screaming Frog names 100,000 to 1 million URLs for the default setting. |
| 500,000 to 2 million URLs | searchVIU, Lumar, Botify, Oncrawl, JetOctopus, Audisto. On the desktop, this works with the right hardware. | Screaming Frog recommends the SEO Spider for crawls below 2 million URLs. The JetOctopus Pro plan includes 2 million URLs per month. |
| over 2 million URLs | searchVIU, Lumar, Botify, Oncrawl, JetOctopus, Audisto | Botify names monthly allowances of up to 50 million pages in the Enterprise plan, Audisto 20 million HTML pages or more, JetOctopus over 100 million pages. With searchVIU, several million URLs a week are no problem, and there is no upper limit. |
Look at the allowance at every size. If you crawl a website with 2 million URLs every week, you need around 8.7 million fetches per month. With rendering, that counts double or triple, depending on the vendor.