Why does the SEO crawler come under suspicion after a migration?

The typical pattern

The migration goes live in the early afternoon, and the manual tests look good. In the evening, the crawl starts for the check. Shortly afterwards the website slows down, and the SEO crawler is clearly visible in the log.

The shared time window leads to the wrong conclusion. Migration, crawl and problems fall on the same day, so the crawl must be to blame. In our experience from many migrations, the crawl comes under suspicion in most cases, regardless of the speed and size of the website. We have not counted this systematically. Often more than an hour is lost. Teams search in the wrong place for days, and the actual post-go-live check stalls.

Why does this happen after migrations in particular?

On a well-established website, an SEO crawl creates hardly any noticeable load. The infrastructure is tuned, the caches are warm and the traffic patterns are known. After a migration, a lot is new, from the CMS to hosting and CDN to templates and security rules. The crawl is often the first traffic that also fetches rarely visited pages and uncached paths. It checks whether redirects are correct, whether pages respond with 200, whether internal links, canonicals, hreflang and structured data are intact and whether load times are acceptable.

1. Capacity does not match production

Staging environments are smaller and rarely reflect real traffic. Load tests simulate people with sessions, pauses and visits to popular pages. Bots fetch without pausing, bypass caches and go deep into rarely visited pages. According to Imperva's 2025 Bad Bot Report, more than half of web traffic in 2024 was automated, namely 51%. Malicious bots accounted for 37%. The figures come from a bot protection vendor. The methodology is not described in the blog post.

• Database queries that are fast with a few records time out with millions of records.
• The number of database connections is set too low for the real load.
• How the CDN handles cache misses and outages can hardly be tested on staging.
• External services only hit their limits under production load.

2. Code changes slow things down under load

Many teams use the migration to refactor code and update dependencies. This creates problems that only show up under load. These include too many individual database queries per page, loops that become too slow with thousands of records, new scripts and tracking pixels, and templates that are more expensive to render. In manual tests with a few pages, this stays invisible. The crawl then fetches archives, categories and rare parameter combinations for the first time.

3. Bot traffic nobody planned for

The old infrastructure had tuned rules for bots. On the new one, default rules often apply, which either block legitimate crawlers or let aggressive scraping through. Monitoring has no baseline yet, so it is hard to say what is normal. At the same time, scanners deliberately look for typical migration errors such as open staging URLs, missing security headers or reachable admin areas.

Why does the SEO crawler stand out first?

A professional crawler identifies itself. According to the HTTP standard, the user agent is self-reported by the client, and malicious bots like to pass themselves off as Chrome or Firefox there.

In our experience, the SEO crawler usually accounts for only 2 to 5% of total traffic in the time window of such an incident. This, too, is a value from project experience and not a systematic measurement.

When the call comes

If the infrastructure team gets in touch, stop the crawl immediately. That is not an admission of fault. You separate the variables and show willingness to cooperate. After that, five questions help with the diagnosis.

1. What share of all requests did the crawler have? “1,000 requests per minute” sounds like a lot. “2.3% of traffic” puts the number into context.
2. How many requests were for HTML and how many for assets? A rendering crawler also loads images, CSS, JavaScript and fonts. Load on the origin comes mainly from the HTML requests; assets usually come from the CDN.
3. When exactly did the problems start? Often the start was before the crawl or between go-live and the crawl.
4. Have the problems persisted since the crawl stopped? Then it was not the cause.
5. How is traffic distributed across all user agents? Looking at all bots shows more than “crawler vs. the rest”.

Segmenting the log correctly

Usually one of these causes then emerges. CDN rules were not carried over properly, and traffic bypasses the cache. The database has too few connections. AI bots or distributed scrapers create a lot of load. Sometimes new code only slows down with real data volumes. How to identify bots in the log is explained in the log file analysis guide.

Aligning before go-live

Weaknesses in a new infrastructure can hardly be ruled out in advance. But you can avoid the crawler getting the blame. Clarify these points with the infrastructure team beforehand.

• A written crawl plan with start time, targets, duration and the expected number of requests.
• IP addresses, user agent and maximum rate, so the team can allow and monitor the crawler.
• Contacts on both sides and thresholds at which the crawl is throttled or paused.
• A short note that the crawler stands out in the log but only makes up a small share of bot traffic.
• Agreed diagnostic steps for an emergency: check shares, HTML vs. assets and origin before anyone names a cause.

Testing bot load on staging

Classic load tests usually leave out bots. So let the crawler run on staging at a much higher rate. Around 20 times normal operation has proven useful. Together with the load test for users, this reveals bottlenecks in capacity, cache and database before go-live. On staging the crawl can be tough; in production it stays gentle. More on this in the SEO check before go-live guide.

In the week of the migration

1. Wait a few hours after go-live until caches are warm and monitoring has its first baseline values.
2. Start at a reduced rate, even if the old website could handle more. Agree the speed with the infrastructure team in advance.
3. Watch response times and throttle as soon as they rise.
4. For large websites, crawl in sections, starting with the most important areas.

Documenting afterwards

After an incident, record when the migration went live, when the crawl started and stopped, and when the problems began and ended. Add the crawler's share of traffic, the actual cause and the measure that fixed it. This note protects the next crawl and builds knowledge within the team.

What does a considerate crawler do?

AI bots and robots.txt

According to OpenAI and Anthropic, GPTBot and ClaudeBot follow robots.txt. Anthropic also supports Crawl-delay (as of 7 April 2026). OpenAI describes how to control GPTBot and OAI-SearchBot via robots.txt and publishes the IP ranges. According to OpenAI, the rules may not apply to ChatGPT-User, which fetches pages on behalf of a user. Load from AI bots comes mainly from volume and frequent fetches. You check whether a bot is genuine via the published IP ranges. More on this in the AI crawlers guide.

Common mistakes

• Starting the crawl at full rate directly after go-live.
• Comparing absolute request numbers instead of shares of traffic.
• Counting a rendering crawler's asset requests as load on the origin.
• Running load tests with simulated users only.
• Only sharing the crawler's IP addresses and user agent after the first incident.
• Not checking after the stop whether the problems persist.

FAQ