We clicked into a single sampled Cloudflare security event to see who our new bot protection was actually catching, and the top blocked source was a documented Googlebot address. Cloudflare’s own managed “Manage AI bots” ruleset was blocking Google’s crawler on our Pro plan, and we would not have noticed until the rankings dropped weeks later.

A search engine you accidentally lock out fails quietly. There is no 500, no alert, no angry customer email. Googlebot gets a 403, treats it as a transient error for a while, and then slowly stops coming back. By the time you feel it, you are reading a traffic chart that bends down and trying to remember what you changed a month ago. The feedback loop is measured in weeks, which makes this an expensive bug.

Green dashboard, wrong conclusion

We had just put Cloudflare in front of a legacy Windows/IIS app. Orange cloud on the apex, www, and media hostnames, then bot protection on, because that is the reason to put a CDN in front of an app you cannot easily rewrite. Managed rules on, Super Bot Fight Mode available, everything green. We took that as verification and moved on. That was our mistake. A green dashboard told us the settings were saved and nothing about what they did to traffic.

Then we went to verify the way this site requires, by reading the events instead of the summary. We drilled into one blocked request and read the fields off it:

  • Service: Managed rules
  • Action: Block
  • Rule: Manage AI bots
  • IP: a sampled source address from Google’s crawler ranges
  • ASN: AS15169, Google LLC
  • User-Agent: Googlebot/2.1

That was the real Googlebot, from a real Google ASN, blocked by a rule Cloudflare ships on by default. We had not written it. On Pro you cannot reconfigure the managed ruleset. You can only add exceptions on top of it.

The fix goes on top

The first instinct is to loosen the managed ruleset so it stops eating crawlers. On Pro you cannot, and the ruleset is also catching real junk. So we added a higher-priority rule that tells Cloudflare to skip its protections for traffic it has already verified as a good bot.

The rule is one line:

  • Expression: (cf.client.bot)
  • Action: Skip
  • Skip: All managed rules, and All Super Bot Fight Mode rules
  • Logging: on

The load-bearing detail is cf.client.bot. It is Cloudflare’s own cryptographically verified good-bot signal, built from reverse-DNS and ASN validation against the published crawler ranges, not a User-Agent match. A scraper that forges Googlebot/2.1 in its User-Agent does not match cf.client.bot, because it cannot pass the reverse-DNS check back to Google’s infrastructure. Real Google, Bing, and the rest get waved through, and spoofers get nothing.

If you have written a bot allow-list keyed on the User-Agent string, this replaces it. The UA is a self-declared label that anyone can type. The verified-bot signal means the provider did the identity check for you.

One Skip rule covers both layers

We were about to flip Super Bot Fight Mode toward Block and turn the geo-block aggression up. A single high-priority verified-bot Skip rule covers both SBFM and the managed rules at once. Before anything gets more aggressive, the crawlers we depend on are already carved out at the top of the stack. Every later tightening has a smaller blast radius, because the thing we cannot afford to break is protected by a rule that runs first.

The geo rule needed the same carve-out. We block everything outside the US and Canada on this site, since 100 percent of customers are in the US and Canada. A plain country block kills Googlebot crawling from a foreign datacenter, so the (not cf.client.bot) clause has to live in that expression too:

(not ip.geoip.country in {"US" "CA"}) and (not cf.client.bot)

Without that second clause, the geo rule deindexes you the same way the managed ruleset did, from a different direction. Every rule that can return a Block needs the clause.

Checking the rule against live events

Deploying the rule was not the finish line. We had caught the original problem by refusing to trust the summary, so we did not trust our own rule on faith either. We deployed, waited, and went back into the events.

This time the Google ranges were passing. A sampled address showed Action: Skip, and behind it two additional addresses from Google’s documented crawler ranges, all skipping. The events summary grew a brand-new “Skip: 156” line within about six minutes, and the 24-hour skip count climbed through the day: 940, then 1.95k, then 2.05k, and it was logging 8.44k verified-bot skips per 24 hours by end of day. A rule that exists in the config and a rule that is catching live traffic are two different claims, and the climbing number is what supports the second one.

The events also showed how hard it gets to say why something blocked once the layers overlap. A US probe we expected to pass came back 403, and it turned out to be the empty-User-Agent rule, not geo. With managed rules, SBFM, a custom geo block and a UA rule stacked, any given block could come from any of them. The Skip rule at the top, with logging on, doubles as an audit trail: you can confirm the good bots get through before the bad-bot rules ever see them.

So before you tighten anything, deploy a Skip rule keyed on cf.client.bot, put (not cf.client.bot) in every rule that can return a Block, and then read your live security events until you see real Google IPs land on Skip. Until you have seen the rule catch actual traffic, it is unverified.