Log file analysis shows what bots really crawl

What is in a log file?

An access log records every request to the server in one line. Most web servers use the Combined Log Format or a variant of it. This is what a line for a Googlebot request looks like. We have shortened the user agent.

Many servers and CDNs also log the response time and the host. The response time shows how fast the server serves bots. You need the host if several domains or subdomains run on the same server.

Identifying and verifying bots

Anyone can set the user agent. If you analyze Googlebot in your logs without checking the IP, you count scrapers too. Google recommends two methods.

1. Reverse DNS. You look up the host name for the IP. It must end in googlebot.com, google.com or googleusercontent.com. Then you use forward DNS to check that the host name points back to the same IP.
2. IP lists. Google publishes the ranges as JSON, for Googlebot in the file common-crawlers.json. There are separate lists for user-triggered fetches.

This is what the check looks like in the terminal.

Which AI bots exist and what they crawl for is explained in the guide AI crawlers.

What does log file analysis show?

The analysis is most useful when you combine it with a crawl and the clicks from Search Console. You then see whether the pages with clicks are crawled often and where the rest of the crawl goes.

Crawl budget

Google describes crawl budget as the set of URLs that Googlebot can and wants to crawl. It has two parts. The crawl capacity limit determines how much load Google puts on your server. If the server responds with 5xx errors or status 429, the limit drops and Google crawls less. Crawl demand depends on the size of the website, the popularity of the URLs and how often content changes.

Google itself calls these figures a rough estimate. According to Google, for smaller websites an up-to-date sitemap and a look at the page indexing report are enough.

Filter URLs: what actually saves crawls?

Filters and sort options in a shop quickly create more URLs than there are products. According to Google, faceted navigation based on URL parameters can generate infinite URL spaces. What matters is whether a method saves crawls or only controls indexing. Your logs show you directly. If Googlebot keeps requesting the filter URLs after the change, the method still costs crawls.

Google calls canonical and nofollow less effective in the long term than robots.txt or fragments. noindex helps with indexing but does not save crawls.

The PRG pattern

Post/Redirect/Get comes from form handling and prevents a form from being submitted twice. For filters, it means the filter options are buttons in a form instead of links. When someone picks a filter, a POST request goes to the server. The server responds with a redirect to the page with the filtered results.

Google can generally only crawl a link if it is an <a> element with an href attribute. A filter button therefore gives Googlebot no URL to follow, while the filter works as usual for users. Google does not describe PRG as a method for filters itself. The effect follows from the rule for crawlable links, and you can check it in your logs. If the filter URLs no longer show up there after the switch, the pattern is working.

Two things go with it. Filters that should rank as landing pages of their own, such as “red sneakers”, still need a normal, linked URL. The result URL after the redirect stays reachable. If it is in the sitemap or linked from elsewhere, Google will still find it.

Search Console crawl stats or log files?

The Crawl Stats report in Search Console is available without any effort and a good place to start. It shows Google's requests, the response time, the host status and a breakdown by status code, file type, purpose and Googlebot type. If the curve drops, that is often the reason to look at the logs.

Getting the logs

1. Find out from IT or your hosting provider where the access logs are. That is the web server, the load balancer or the CDN.
2. If you use a CDN such as Cloudflare, its logs are part of it. Requests that the CDN answers from the cache never reach the server and are missing from its logs.
3. Decide how long the logs are kept. For a comparison before and after a release or relaunch, you need the weeks before.
4. Filter for bots early. For SEO, the lines from search engines and AI bots are enough.
5. Clarify data protection. IP addresses can be personal data (CJEU, judgment of 19 October 2016, C-582/14, Breyer). No legal advice.

Common mistakes

• Identifying Googlebot only by its user agent.
• Only analyzing the origin server's logs, even though a CDN sits in front of it.
• Looking at a single day only. Crawl patterns only show up over several weeks.
• Setting pages to noindex to save crawl budget.
• Only looking for the logs once the crawl has already dropped. Then there is nothing to compare with.

FAQ