Bot protection that lets customers, Googlebot and AI bots through

More bots than humans

The dilemma starts with a simple number. According to Imperva's 2026 Bad Bot Report, 53 % of all web traffic in 2025 came from automated programs and 40 % from bad bots. The year before, the figures were 51 % and 37 %. The task for website owners is therefore to stop abuse without driving away real users and wanted bots.

The source is Imperva (Thales), Bad Bot Report 2025 and 2026. Imperva does not state the share of good bots explicitly; it is the remainder. The share varies by industry. For travel websites, Imperva's blog on the 2025 report put the share of bad bots at 48 %, while the press release from the same day says 41 %.

What do bad bots cost?

Bad bots scrape prices, test stolen credentials (credential stuffing), take over customer accounts and put load on servers. Reliable cost figures come almost exclusively from vendors of bot defense. A white paper by F5 from March 2025 works through an example. A large online retailer with 31 million customer accounts and 54 US dollars of monthly revenue per account loses an estimated one million US dollars a year to bot attacks. F5 used its own calculator to model the benefit after switching to modern bot defense.

These are a vendor's projections for its own product. What is interesting is where the largest item comes from. Most of the gain comes from less friction for real customers, in other words from a defense that lets the right people through.

AI crawlers as a new load

AI services have added a new kind of bot traffic. According to OpenAI, ChatGPT had 100 million weekly users in November 2023, 400 million in February 2025, more than 800 million in October 2025 and 900 million in February 2026. The crawlers behind these services fetch many pages and send few visitors back. Cloudflare measures this as the ratio of crawls to referrals.

Cloudflare itself points out that the Claude app sends no referrer, so the figures for Anthropic may be too high. Current figures are on Cloudflare Radar under AI Insights.

Read the Docs described in July 2024 how expensive this can get for small websites. A single crawler downloaded 73 TB of zipped HTML files in May 2024, almost 10 TB in one day, and caused more than 5,000 US dollars in bandwidth charges. After the team blocked AI crawlers, bandwidth fell by 75 %, from around 800 GB to around 200 GB a day. Had all this traffic hit its own servers, it would have cost around 50 US dollars a day according to Read the Docs, or about 1,500 US dollars a month.

Every website owner therefore faces a choice between blocking AI crawlers and being missing from AI answers, or allowing them and carrying the load. There is a third way, described further down in the section on attack surface.

When defense hits the wrong people

In the fight against bots, many websites become their own opponent. Every hurdle that stops a bot also costs a share of real visitors. CAPTCHAs show this most clearly.

A Fastly survey of 881 US online shoppers found that 18.7 % give up after a failed CAPTCHA. This is also a vendor survey, but it points in the same direction as the studies.

VPN users and corporate networks

Many people browse via a VPN, for privacy reasons or because their employer requires it. According to Security.org, 42 % of US adults used a VPN in 2026, up from 32 % in 2025. These users share a few IP addresses with many others, including attackers. Cloudflare writes that visitors from shared networks, VPNs and Tor often have a poor IP reputation and are therefore more likely to see a challenge page. The same applies to corporate gateways and mobile networks that bundle many users behind one address. What is meant to keep bots out then keeps real customers out too.

What does bot protection mean for SEO and GEO?

For search engines and AI services, bot defense is a silent source of errors. robots.txt may allow a bot while the firewall or the CDN's bot protection turns it away with a challenge page or status 403. Googlebot then gets a check page instead of your page, and AI services cannot fetch your content.

• If you need less crawling for a short time, Google says to send 500, 503 or 429 instead of 200, ideally with Retry-After. Google warns that this should not last longer than one or two days, otherwise the website suffers in Google products.
• According to Google, 403 and 404 are unsuitable for throttling. Google treats them as the answer for the URL, with consequences for indexing.
• Check in your log files whether Googlebot and the AI bots you want to allow get status 200. The guide Log file analysis shows how.

AI bots and Cloudflare's defaults

Many websites sit behind Cloudflare, and its defaults for AI bots have changed twice. On July 1, 2025, Cloudflare announced that it would block AI crawlers by default for new domains unless they pay content creators. Since July 1, 2026, Cloudflare splits AI traffic into three classes: Search, Agent and Training. For each class you choose whether it is blocked on all pages, only on pages with ads, or not at all.

From September 15, 2026, new defaults apply to domains joining Cloudflare. Bots in the Training and Agent classes are blocked on pages with ads, while Search remains allowed. Crawlers that combine search and training fall under the Training block. The announcement does not say whether and how existing domains are affected, so check the settings yourself.

Cloudflare also offers a managed robots.txt with Content Signals, which state whether content may be used for search (search), as input for AI answers (ai-input) or for training (ai-train). Cloudflare's default is search=yes, ai-train=no. On October 30, 2026, Google announced that it will support a shared training token in robots.txt, which is currently being drafted at the IETF. The guide AI crawlers covers which AI crawlers exist and how to control them via robots.txt, and the guide Cloudflare for SEO and GEO covers the Cloudflare settings.

Telling real bots from fake ones

Anyone can set a user agent, which is why many attackers pretend to be Googlebot. A safe bot rule only lets a bot through if it identifies itself in one of these three ways.

1. Published IP lists. Since March 2026, Google publishes its lists at developers.google.com/static/crawling/ipranges/, split into crawlers, special crawlers, user-triggered fetchers and agents. Bing publishes bingbot.json, OpenAI publishes lists for GPTBot, OAI-SearchBot and ChatGPT-User, and Perplexity for PerplexityBot and Perplexity-User.
2. Reverse DNS. Google recommends resolving the IP in reverse, checking the domain (googlebot.com, google.com or googleusercontent.com) and then resolving it forward. The request is genuine only if both directions lead to the same IP.
3. Web Bot Auth signatures. Bots sign their requests with HTTP Message Signatures under RFC 9421. Cloudflare has listed such signed agents as Verified Bots since July 1, 2026. Google is testing Web Bot Auth for some AI agents, and the ChatGPT cloud browser signs with the header Signature-Agent: "https://chatgpt.com". An IETF working group is developing the standard.

This is how you check an IP from your log via reverse DNS.

On Cloudflare, bot management performs this check for all bots on the Verified Bots list. A custom rule that blocks or allows everything with “bot” in the user agent bypasses it.

Graduated responses, from throttling to blocking

Good bot defense distinguishes first and decides second. Three principles help.

1. Analyze behavior. Modern systems assess the behavior and context of a request. Cloudflare assigns a bot score for this.
2. Respond in stages. Throttle suspicious requests, serve them cached content or challenge them before you block. Let the bots wait and the users through.
3. Shrink the attack surface. The fewer unnecessary URLs your website has, the less good and bad bots find to fetch. More on this in the next section.

For users on shared networks, Privacy Pass also lets a browser prove that a human is present without identifying the person.

Using CAPTCHAs accessibly

Since June 28, 2025, Germany's Barrierefreiheitsstärkungsgesetz (BFSG) has applied, implementing EU Directive 2019/882 (European Accessibility Act). It covers many online shops, among others. The usual benchmark for websites is WCAG. Success criterion 3.3.8 of WCAG 2.2 (level AA) requires that logging in does not depend on a cognitive test such as retyping distorted characters, unless there is an alternative or help. Recognizing objects is exempt. The W3C note “Inaccessibility of CAPTCHA” explains why classic CAPTCHAs exclude people with disabilities. Cloudflare states that Turnstile meets WCAG 2.2 AA. This section is not legal advice.

Fewer URLs, less attack surface

While many websites keep building higher walls, the opposite approach often helps. Make your website less rewarding for crawlers by clearing out the maze of unnecessary URLs in which good and bad bots alike waste resources.

Google writes that websites with fewer than a few thousand URLs are usually crawled efficiently without intervention. According to Google, crawl budget becomes relevant for large websites with more than a million pages that change about weekly, and for medium-sized websites with more than 10,000 pages that change daily. Google warns that its crawlers may not explore the rest of a website if they spend too much time on URLs they should not crawl.

When filters create millions of URLs

According to Google, faceted navigation is by far the most common source of overcrawling reported by website owners. A category page with five filters such as color, size, price, availability and sort order can create thousands of separate URLs. Builtvisible describes the case of Very.co.uk with more than 11.2 million indexed URLs for around 160,000 actual pages. After the clean-up, organic revenue rose by 325 % and organic traffic by 220 %, measured against August 2012.

Why do noindex and canonical barely reduce crawling?

noindex and canonical control what goes into the index. To read them, however, a bot has to fetch the page and uses bandwidth and server capacity in the process. In its crawl budget documentation, Google explicitly says that noindex is unsuitable for this because Google still requests the page. Google also advises against using robots.txt only temporarily to reallocate crawl budget.

What helps in practice?

• PRG pattern for filters. Post/Redirect/Get was originally developed to prevent duplicate form submissions. Filters run through a POST form, and the server redirects to the filtered page. Because crawlers generally do not follow POST forms, no crawlable filter URLs are created. This is a practice from the SEO community, described for example in the Seobility wiki. Google has not recommended it.
• Block filters via robots.txt, for example with wildcards for entire parameter patterns. Google mentions this in its faceted navigation documentation, as well as URL fragments (#) for filters and status 404 for empty filter combinations.
• Clean up internal links. Remove links to sorted or filtered variants that nobody needs to find.
• Use parameters sparingly. Avoid session IDs and tracking parameters in internal links. The Search Console URL parameters tool no longer exists.
• Prune content. Delete or merge pages that bring no traffic for months, and redirect them.

What have others achieved?

Greek price comparison site Skroutz cut its indexed URLs from 25 million to 7.6 million in 2018 and 2019. The team set 2.2 million internal search pages to noindex and redirected 300,000 URLs to matching categories. New URLs then reached the index within a few days instead of two to three months. In the Conductor Academy, Conductor describes how the average crawl budget of its own website rose from 27 to 253 pages a day within two years.

Research also shows how much crawling is wasted. According to the 2025 Crawl4LLM study, existing work often discards more than 90 % of the raw data collected. The authors' approach reaches more than 95 % of the best possible performance while crawling only 2.2 % of the data. At the same time, more and more websites restrict access. In July 2024, the Data Provenance Initiative found that more than 5 % of tokens in the C4 dataset and more than 28 % of its most actively maintained, critical sources were fully restricted via robots.txt. Through terms of service, as much as 45 % was restricted.

How to measure success

• Crawl efficiency. Share of crawls on important pages out of all crawls, from your log files.
• Bot share of bandwidth. Bandwidth of bot traffic relative to total bandwidth.
• Error rate. Share of 4xx and 5xx responses to bots, from the logs.
• Ratio of URLs to pages. All discoverable URLs relative to the actual content pages, from a full crawl.
• Search Console crawl stats. Requests per day, response time and responses by status code.

Typical steps by system

• Online shops. Block filter parameters or move them to the PRG pattern, set canonicals for product variants, handle pagination cleanly.
• WordPress. Remove unnecessary archives, feeds and parameter URLs, use your SEO plugin's crawl optimization.
• Enterprise CMS. Custom robots.txt rules, rebuild faceted navigation, remove session IDs from URLs.

How to proceed

1. Review the bot and AI settings of your CDN and firewall, including the defaults that apply without any action on your part.
2. Compare them with your robots.txt. Both should want the same thing.
3. Check in the log files which status Googlebot, Bingbot and the AI bots you want receive. Challenge pages and 403 responses to wanted bots are errors.
4. Only let bots through after verification via IP list, reverse DNS or signature.
5. Wherever possible, replace hard blocks with throttling and unobtrusive challenges. Use CAPTCHAs only at sensitive points and offer accessible alternatives.
6. Besides blocked bots, also measure false positives, meaning real users and wanted bots that get challenged or blocked.
7. Remove unnecessary URLs, especially from filters and parameters.
8. Review the settings regularly. Bots, providers and defaults change constantly, and some bots explicitly deserve a seat at the table.

Common mistakes

• robots.txt allows a bot, but the CDN blocks it anyway.
• A rule lets through everything with “Googlebot” in the user agent and opens the door to attackers.
• Data center IPs are blocked across the board and hit the cloud agents of real customers. The guide Optimize your website for AI browsers explains what AI browsers need.
• 403 or 404 is sent for throttling instead of a temporary 429 or 503.
• A CAPTCHA on every product page instead of only at login and checkout.
• Filters create millions of URLs, and the team tries to manage them with noindex and canonical alone.

Frequently asked questions