Technical SEO

How do you crawl a website for a technical SEO analysis?

Seven steps from the goal to a prioritized list. Plus the crawlers that fit the job and the cross-check with Search Console and log files.

200301404
Michael WeberLast updated: November 16, 202615 min read
In short
  • An SEO crawler fetches your website page by page, follows the internal links and stores what search engines find at every URL.
  • The process has seven steps. You define the goal, choose the crawler, configure it, crawl, analyze, prioritize and repeat the crawl.
  • Desktop crawlers such as Screaming Frog and Sitebulb run on your computer. Cloud crawlers such as Lumar, Botify, Oncrawl, JetOctopus and searchVIU crawl on the vendor's servers and on a schedule.
  • The crawl shows what is on your website. What Google makes of it is shown in Search Console and in your server's log files.

What is an SEO crawler and how does it work?

An SEO crawler is a program that fetches a website the way a search engine crawler does. It starts at one or more URLs, reads the source code, collects the links in it and fetches those pages next, until it knows every reachable URL. For every URL, it stores the status code, redirect target, title, canonical, robots directives, headings, links and other data.

Many crawlers also load every page in a browser and execute the JavaScript. This way, the crawl also captures content and links that only appear in the browser. Google works in a similar way. According to Google's In-depth guide to how Google Search works, Google discovers new pages through links from known pages and through submitted sitemaps, and renders them using a recent version of Chrome.

Googlebot crawls the web for Google's index. An SEO crawler crawls your website on your behalf and gives you the data as a table, report or export. Whether Google actually fetches and indexes a page is shown in Search Console and in your log files. More on this in the section Checking crawling and indexing.

Crawling a website in seven steps

The seven steps apply to every crawler, from the desktop tool to the cloud platform.

1. Define the goal

The goal determines what you crawl and with which settings. For a full inventory, you need the entire website. If clicks drop for one page type, the affected directory is often enough. Before a relaunch, you crawl the test environment and compare it with the live website.

Write down beforehand which hosts and subdomains belong to the site, which page types exist and which questions the crawl should answer. What else an audit covers is explained in the guide SEO audit.

2. Choose a crawler

There are two types. A desktop crawler is a program on your computer. How many URLs it can handle depends on memory and disk space, and the computer has to keep running during the crawl. A cloud crawler runs on the vendor's servers. It crawls on a schedule, and your team works with the same data. The table shows what seven crawlers stand for, according to their vendors.

CrawlerTypeWhat it stands for (according to the vendor)
Screaming Frog SEO SpiderDesktop for Windows, macOS and LinuxAudits for over 300 SEO issues. It crawls 500 URLs without a license and is recommended for crawls below 2 million URLs. JavaScript rendering, crawl comparison and custom extraction require a license.
SitebulbDesktop and cloudAudits with prioritized recommendations (“Hints”) and visualizations. Sitebulb Desktop has a default limit of 500,000 URLs per audit, which can be raised to 2 million. For Sitebulb Cloud, the vendor states 10 million URLs per audit.
Lumar (formerly Deepcrawl)CloudStates up to 450 URLs per second without rendering and 350 with rendering, measured in a 2022 test. Crawl speed and time windows can be configured.
BotifyCloudPlatform for very large websites. The SiteCrawler covers millions of pages and renders with the same engine as Googlebot. The LogAnalyzer analyzes log files.
OncrawlCloudCrawls millions of URLs with JavaScript rendering, without artificial limits on speed, depth or site size. Also analyzes log files.
JetOctopusCloudCrawl, log files and Search Console data in one tool. The crawler handles up to 250 pages per second, and you choose the speed through the number of threads. Prices are listed on the website.
searchVIUCloudRenders every URL in Chrome and runs a full crawl of the website every week. Every URL goes through over 100 checks. New problems arrive as alerts, and Page Watch checks your most important pages between crawls. Project data is stored in the EU.

The information comes from the vendors' websites, retrieved on November 16, 2026. SEO suites such as Ahrefs and Semrush also include a crawler, “Site Audit”, which according to the vendors crawls the entire website and checks it for technical and on-page issues. Four typical cases help you choose.

  • One-off analysis of a small or medium-sized website. Screaming Frog and Sitebulb on your own computer are a good fit.
  • Large website with hundreds of thousands or millions of URLs. Lumar, Botify, Oncrawl, JetOctopus, Sitebulb Cloud and searchVIU are the options here. How cloud crawlers for large websites differ is shown on the page Enterprise SEO crawlers compared.
  • Crawl and log files in one tool. According to the vendors, Botify, Oncrawl and JetOctopus also analyze log files. At searchVIU, log file analysis is an add-on to SEO Monitoring.
  • Regular crawls and monitoring. According to the vendors, Screaming Frog, Lumar and Semrush also offer scheduled crawls. searchVIU runs a full crawl every week, and Page Watch checks your most important pages continuously, throughout the day.

How desktop and cloud crawlers differ in day-to-day work is shown in the comparison Screaming Frog alternative. Where vendors store your crawl data is covered in SEO crawlers with data in the EU.

3. Configure the crawl

Small websites can be crawled with the default settings. For large websites, clarify these points before you start.

  • Start URLs and scope. Define where the crawl starts, which hosts and directories belong to it and how the crawler handles URL parameters.
  • Additional URL sources. Through links, a crawler only finds linked pages. With XML sitemaps, URLs from Search Console and your own URL lists, pages without an internal link show up too. The guide to finding orphan pages shows how to track them down.
  • JavaScript rendering. If content or links only appear in the browser, the crawler has to render. According to Google's JavaScript SEO basics, Google processes JavaScript web apps in the three main phases of crawling, rendering and indexing. How Google handles each phase is explained in the guide to crawling, rendering and indexing JavaScript. By default, Screaming Frog only crawls the source code. With rendering, the crawl takes longer because the crawler loads every page including its resources. More on this on the JavaScript SEO page.
  • User agent and device. Crawl as a smartphone. According to Google's mobile-first indexing best practices, Google uses the mobile version of a site's content for indexing and ranking. Bot protection and CDNs often block unknown crawlers. Your IT team can allowlist a dedicated user agent and a fixed IP. searchVIU crawls as searchVIUbot from the fixed IP 35.190.196.200. More on this in the guide Bot protection and SEO.
  • Crawl rate. Agree on timing and crawl rate with your IT team. Screaming Frog recommends this too, and its crawler runs with 5 threads by default. Oncrawl names 2 to 5 URLs per second as a good rate for most sites. searchVIU sets the number of concurrent requests per project and agrees on it with your IT team.
  • robots.txt. To see what search engines see, the crawler follows robots.txt. According to Google, the file tells crawlers which URLs they can access, and “it is not a mechanism for keeping a web page out of Google.”
  • Access. A test environment behind password protection or an IP block needs login credentials or allowlisting.

4. Crawl

Start with a test run of a few hundred URLs. Check whether the status codes make sense, whether the rendered page is complete and whether the crawler runs into a trap, such as calendars, filter combinations or session parameters. Many responses with 403 or 429 point to bot protection.

Then comes the full run. Keep an eye on response times and the error rate, and lower the crawl rate if the server slows down. How long the crawl takes depends on the number of URLs, the crawl rate and rendering. Screaming Frog calculates that a website with 1 million pages takes over two days at 5 URLs per second. For a worked example with searchVIU, see the use case Crawling large websites.

5. Analyze

Analyze the results by page type. An error in a template affects every page of that type, so it is a single task. These areas belong in every technical analysis.

AreaWhat you check in the crawlWhat Google says
Status codes4xx and 5xx, internal links to error pagesGoogle doesn't index URLs that return a 4xx status code. 5xx and 429 errors prompt Google's crawlers to temporarily slow down.
RedirectsChains, loops, internal links to redirecting URLsBy default, Google's crawlers follow up to 10 redirect hops.
Indexabilitynoindex in the HTML and in the HTTP header, blocks in robots.txtFor noindex to work, the page must not be blocked by robots.txt.
CanonicalsMissing tag, target redirects or is not indexable, conflict with hreflangRedirects and rel="canonical" are strong signals, sitemap inclusion is a weak one.
Title and descriptionMissing, duplicate or identical across an entire page typeEvery page should have a title. Google primarily uses the content on the page to create the snippet.
HeadingsMissing H1, the same H1 on many pagesHeading elements such as the H1 are one of the sources Google uses to create title links.
Internal links and click depthPages without an internal link, important pages many clicks away from the homepage, links without hrefGoogle can only discover links that are a elements with an href attribute.
hreflangMissing return links, missing self-reference, target not indexableIf two pages don't both point to each other, Google ignores the tags.
Structured dataMissing per page type, syntax errorsGoogle recommends JSON-LD and checking with the Rich Results Test.
Source code vs. rendered HTMLTitle, canonical, robots directives, content and links that JavaScript changesOnce Google's resources allow, a headless Chromium renders the page.

You can check individual pages without a crawl using the free tools, such as the JavaScript Rendering Check, the Redirect Checker or the Hreflang Testing Tool. All searchVIU checks are listed in the public list of checks.

6. Prioritize

A crawl quickly produces thousands of issues. Sort them by the number of affected pages and by those pages' clicks from Search Console. Anything that prevents crawling or indexing of pages with clicks goes to the top. A missing title on three pages without traffic can wait. Every task comes with the cause, the page type, example URLs and the number of affected pages.

7. Repeat and compare

A crawl is a snapshot. After every fix, you crawl again and check whether the change is live. The comparison with the previous crawl shows new error pages, pages that have disappeared and changed titles, canonicals or robots directives. According to the vendor, Screaming Frog compares two crawls in “Database Storage” mode and with a license. Cloud crawlers keep earlier crawls on the server. A fixed schedule makes sense, for example every week or after every release.

How do you extract specific on-page elements?

An SEO crawler captures the title, description, H1, canonical, robots directives, hreflang and structured data on its own. For everything else, there is custom extraction. You write a rule that tells the crawler which part of the page to extract, and you get the value as a separate column for every URL.

Typical cases are the price and availability on a product page, the last breadcrumb item, the date of an article, the number of products on a category page or images without an alt attribute. Here is how to go about it.

  1. Open an example page in the browser, inspect the element and note its XPath or CSS selector.
  2. Create the rule in the crawler and test it on two or three URLs per page type.
  3. Extract from the rendered HTML if the value only reaches the page through JavaScript.
  4. Limit the rule to the right page type, for example with a URL pattern.
  5. Analyze the new column together with status code, indexability and clicks.

The following three XPath expressions show the principle. Adapt them to your website's markup.

# Price on a product page //span[@itemprop="price"]/@content # Last breadcrumb item //nav[@aria-label="Breadcrumb"]//li[last()] # Count images without an alt attribute count(//img[not(@alt)])

According to the vendor, Screaming Frog extracts data using CSS Path, XPath or regex. The feature requires a license. Lumar describes its crawler as “capable of extracting almost any information from your web pages”. searchVIU extracts with XPath and regex from the rendered HTML, including structured values such as filter groups with their values. Every rule becomes a separate column in the crawl and is available in the reports. Page Watch checks on your most important pages whether an extraction suddenly comes back empty. searchVIU support helps you set up the rules.

How do you check how search engines crawl and index your website?

Your own crawl shows what a crawler finds on the website. What Google actually fetches and adds to its index is shown in Search Console and in your server's log files. According to its own documentation, Google doesn't guarantee that it will crawl or index any page. That is why the cross-check is worth it for every important page type.

SourceWhat it showsWhat you use it for
URL Inspection tool in Search ConsoleGoogle's indexed version of a URL with the last crawl, page fetch, crawling and indexing details, and the user-declared and Google-selected canonical. The live test fetches the URL again and shows the HTML and a screenshot.Check a single URL, for example after a fix
Page indexing reportWhich pages Google can find and index, with reasons such as “Crawled - currently not indexed”, “Discovered - currently not indexed” or “Page with redirect”. The list of examples is limited to 1,000 rows.Spot patterns that affect entire page types
Crawl Stats reportTotal crawl requests, total download size and average response time, broken down by response, file type, purpose and Googlebot type. It also shows the host status. The report is available only for root-level properties.Spot server problems and changes in crawling
Sitemaps reportFor each sitemap, the status, “Last read” and the number of “Discovered pages”Check whether Google reads your sitemaps
Server log filesEvery bot request with URL, time, status code and user agentSee which URLs Googlebot and other bots really fetch

The Search Console reports show totals and example URLs. The log files show every single request. In a log file, however, any bot can pose as Googlebot. Google's documentation on how to verify requests from Google crawlers and fetchers describes two ways to spot genuine requests. You run a reverse DNS lookup on the accessing IP address, or you match the IP address against Google's published IP ranges. How to analyze log files is explained in the guide Log file analysis.

The picture becomes meaningful when you compare it with your own crawl. Three questions help.

  • Do all indexable pages from the crawl have impressions in Search Console?
  • Does Googlebot fetch URLs, according to the log files, that are missing from the crawl or not indexable there?
  • Are important page types listed under “Crawled - currently not indexed” in the Page indexing report?

For Bing, Bing Webmaster Tools has its own URL Inspection. The Search Console reports are explained in detail in the guides Google indexing and Crawl errors in Search Console.

What this looks like in searchVIU

searchVIU runs a full crawl of your website every week and renders every URL in Chrome. After every crawl, you know what has changed and which issues affect clicks.

  • Every URL goes through over 100 checks, from status codes to structured data.
  • Topic reports cover canonicals, hreflang, JavaScript rendering, structured data and internal links.
  • The SEO Issues cockpit holds your to-do list per page type, weighted by affected pages and Google clicks.
  • The comparison with the previous crawl shows what is new and what has been fixed.
  • New problems reach you as alerts by email, Slack, Teams or webhook.
  • Between crawls, Page Watch checks your most important pages continuously, throughout the day.

searchVIU crawls websites with millions of URLs and renders every single page in Chrome. Several million URLs a week are no problem. How the crawler works is shown on the SEO Crawler page. The to-do list in detail is shown on the SEO Audit Tool page.

FAQ

What is an SEO crawler and how does it work?

An SEO crawler is a program that fetches a website the way a search engine crawler does. It starts at a URL, follows the links and stores data for every page, such as status code, title, canonical, robots directives and links. Many crawlers also render the page in a browser so that content from JavaScript is captured too.

How do I crawl a website for technical SEO analysis?

In seven steps. You define the goal, choose a crawler, configure start URLs, rendering, user agent and crawl rate, run the crawl, analyze the results by page type, prioritize by affected pages and clicks and crawl again after the fixes.

How can you use an SEO crawler to optimize a website?

The crawl shows where pages return errors or are excluded from indexing, where redirects and canonicals are set incorrectly and which pages have too few internal links. This gives you a list of tasks per page type. The next crawl shows whether the errors are fixed.

Which SEO crawler suits which website?

For one-off analyses of small and medium-sized websites, desktop crawlers such as Screaming Frog and Sitebulb are a good fit. For large websites and regular crawls, cloud crawlers such as Lumar, Botify, Oncrawl, JetOctopus and searchVIU are the options. searchVIU is the right fit when every URL needs to be rendered and you need alerts between crawls.

How can I crawl a website to analyze specific on-page SEO elements?

Every SEO crawler delivers standard elements such as title, description, H1 and canonical. You extract other elements with custom extraction, using a rule written as XPath, CSS selector or regex. The result appears as a separate column for every URL in the crawl.

How can I check how search engines crawl and index my website?

For Google, you use Search Console. The URL Inspection tool shows the status of a single URL, the Page indexing report shows the reasons why pages are missing and the Crawl Stats report shows Googlebot's requests. Your server's log files show every request from every bot. Your own crawl adds the view of the entire website to both.

Do I need JavaScript rendering when crawling?

Yes, as soon as content, links or elements such as the title and canonical are created or changed by JavaScript. Without rendering, the crawler only sees the source code. Comparing the source code with the rendered HTML shows what JavaScript changes.

Can a crawl overload my server?

Yes, if the crawl rate is too high. Agree on timing and the number of concurrent requests with your IT team and watch the response times. searchVIU halves the rate of the entire crawl as soon as the server responds with HTTP 429.

How often should I crawl my website?

After every major change and on a fixed schedule. searchVIU runs a full crawl of your website every week and checks your most important pages in between with Page Watch.

searchVIU crawls your website every week, renders every URL in Chrome and compares every crawl with the previous one. What the crawler captures and how it behaves on your server is shown on the SEO Crawler page.

Request a demo

Sources

Retrieved on November 16, 2026. If something is out of date or missing, let us know.

  1. Google Search Central, In-depth guide to how Google Search works: https://developers.google.com/search/docs/fundamentals/how-search-works
  2. Google Search Central, Understand JavaScript SEO basics: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
  3. Google Search Central, Mobile-first indexing best practices: https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing
  4. Google Search Central, Introduction to robots.txt: https://developers.google.com/search/docs/crawling-indexing/robots/intro
  5. Google Search Central, Block Search indexing with noindex: https://developers.google.com/search/docs/crawling-indexing/block-indexing
  6. Google Search Central, How to specify a canonical URL: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
  7. Google Search Central, Influencing title links: https://developers.google.com/search/docs/appearance/title-link
  8. Google Crawling Infrastructure, HTTP status codes and network errors: https://developers.google.com/crawling/docs/troubleshooting/http-status-codes
  9. Google Crawling Infrastructure, Verify requests from Google crawlers and fetchers: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
  10. Search Console Help, Page indexing report: https://support.google.com/webmasters/answer/7440203