Cloudflare sits in front of your server and has a say in SEO and GEO

What does Cloudflare mean for SEO?

You may know Cloudflare as a CDN, a security service or a DNS provider. For SEO, what matters most is that Cloudflare sits as a reverse proxy between all traffic and your actual server. According to W3Techs, 27.3 percent of all websites use Cloudflare (as of October 2026).

Think of a bouncer in front of a club. The club is your server, and Cloudflare stands at the door. The bouncer decides who gets in, who is turned away and who has to show ID. If someone is turned away, the club never finds out. This brings speed, protection against DDoS attacks and caching at the edge of the network. But it also creates a blind spot that many SEOs are not aware of.

Whether a website sits behind Cloudflare is shown by our Technology Detection.

Why does this matter even more for GEO?

Besides search engines, AI systems now fetch your content too. Roughly speaking, there are three types of crawlers.

Google's AI Overviews use the regular Googlebot.

This creates a dilemma. If you block all AI crawlers, you protect your content from unsolicited training but miss out on AI answers. If you let them all in, other companies' models train on your content. Your Cloudflare settings help decide which AI answers your brand can appear in at all. For ChatGPT, Perplexity or Gemini, what counts is whether their crawlers are allowed to fetch your pages and are actually able to.

Your server logs are incomplete

In a log file analysis, you evaluate what your server has seen. But your server only sees what Cloudflare lets through. Its logs are missing

• blocked requests from bots, in the worst case including Googlebot,
• requests that received a JavaScript challenge or a CAPTCHA,
• requests that Cloudflare answered entirely from the cache,
• requests answered by a Worker at the edge of the network,
• requests throttled by rate limiting.

Security features that can hit crawlers

Cloudflare maintains a list of verified bots, which includes Googlebot and Bingbot. Custom rules can override this allowance, though. Configuration errors are often only noticed months later.

Caching and what crawlers see

If Cloudflare serves a page from the cache, it may be slightly out of date. After a content change, what matters is whether and when the cache is purged. Our case study on HTML caching with Cloudflare shows what this looks like in an online shop.

Where do you find what in the dashboard?

Many reports depend on the plan, such as Log Explorer or detailed bot analytics. Ask your IT which plan you are on. Without access to the dashboard, Cloudflare Radar shows public data on AI crawlers across the whole network, such as which bots are blocked most often.

Asking IT for access

In most companies, IT is responsible for Cloudflare, including DNS, security rules and infrastructure. You do not need admin rights for this. Cloudflare assigns fine-grained permissions, and read access to a few areas is enough.

1. Be specific. “I need read access to Security Events to check whether Googlebot is being blocked.”
2. Give a reason. A drop in crawling or indexing over the last month is a good reason.
3. Start small. One area first, then more once the benefit is visible.
4. Offer an alternative. If access is not possible, ask for a regular report, for example all Security Events with “Googlebot” in the user agent over the last 30 days.
5. Be involved in changes. Ask to be consulted before new security rules go live.

As an indirect signal, there is still the Crawl Stats report in Search Console. If crawl requests suddenly drop, a block may be behind it.

robots.txt via Cloudflare

Managed robots.txt

With managed robots.txt, Cloudflare manages rules for AI crawlers directly at the edge of the network, on all plans. If a robots.txt already exists, Cloudflare puts its rules in front of it and serves both as one file. If there is none, Cloudflare creates one with Disallow rules for known AI crawlers. According to Cloudflare, more than 3.8 million domains were using this feature in September 2025.

After switching it on, always check the file that is actually served live. If you maintain complex rules, for example per directory, you are better off keeping robots.txt in your own hands.

Content Signals

The Content Signals Policy has existed since September 2025. It extends robots.txt with information on what content may be used for.

With managed robots.txt, Cloudflare sets search=yes and ai-train=no and leaves ai-input open. Content Signals are preferences, not a technical block. If you really want to lock out AI crawlers, you also need rules in AI Crawl Control or the WAF.

Enforcing robots.txt

In December 2024, Cloudflare introduced a feature called Robotcop that shows which AI services comply with your robots.txt. At the push of a button, it turns the rules for AI bots into a firewall rule. Today this is part of AI Crawl Control, on all plans.

Google-Extended and AI Overviews

Google-Extended controls whether Google uses content to train Gemini and for grounding in some other Google services. According to Google, the token has no effect on inclusion in Google Search and no weight in ranking. AI Overviews and AI Mode follow the rules of Search, meaning robots.txt for Googlebot and snippet controls such as nosnippet.

What has changed since then?

On 1 July 2026, Cloudflare split its AI crawler settings into three categories: Search, Agent and Training. According to Cloudflare, multi-purpose crawlers such as Googlebot, Applebot and Bingbot are blocked if you block training. If you wanted to prevent this, you had to opt out in the Security Settings before 15 September 2026. For domains newly added to Cloudflare since 15 September 2026, training and agents are blocked by default on pages with ads.

So check whether a block on AI training is active in your zone and whether Googlebot and Bingbot still get through.

Checklist

• Is the website behind Cloudflare, and which plan is it on?
• Read access to Security Events or a regular report has been agreed.
• No country block and no WAF rule hits Googlebot or Bingbot.
• The AI training setting has been chosen deliberately and its effect on Googlebot checked.
• The robots.txt served live matches what you want.
• Log file analyses use Cloudflare's logs; the server's logs are not enough.

Common mistakes

• Crawl analyses based only on server logs.
• Switching on Bot Fight Mode and then not checking whether search engines get through.
• Blocking AI training across the board without knowing the consequences for multi-purpose crawlers.
• Switching on managed robots.txt and never looking at the combined file.
• Blocking Google-Extended to disappear from AI Overviews.

FAQ