I clicked into a single sampled Cloudflare security event to see who my new bot protection was actually catching, and the top blocked source was a documented Googlebot address. That is Googlebot. Cloudflare’s own managed “Manage AI bots” ruleset was blocking Google’s crawler on my Pro plan, and I would not have noticed until the rankings dropped weeks later.

Here is the thing about a search engine you accidentally lock out: nothing breaks loudly. There is no 500, no alert, no angry customer email. Googlebot gets a 403, treats it as a transient error for a while, and then slowly stops coming back. By the time you feel it, you are reading a traffic chart that bends down and trying to remember what you changed a month ago. This is the most expensive kind of bug because the feedback loop is measured in weeks.

What I was actually doing

I had just put Cloudflare in front of a 20-year-old Windows/IIS/ASP app. Orange cloud on the apex, www, and media hostnames, then turned on the bot protection because that is the whole reason you reach for a CDN in front of a legacy app you cannot easily rewrite. I treated managed rules on, Super Bot Fight Mode available, all green, and the dashboard looking done as sufficient verification, then moved on. That was my mistake. A green dashboard is not proof anything is actually working the way I assumed, and I had just accepted it as if it were.

Then I went to verify, which on this site means actually reading the events instead of trusting the summary. I drilled into one blocked request and read the fields off it:

  • Service: Managed rules
  • Action: Block
  • Rule: Manage AI bots
  • IP: a sampled source address from Google’s crawler ranges
  • ASN: AS15169, Google LLC
  • User-Agent: Googlebot/2.1

That is the real Googlebot, from a real Google ASN, getting blocked by a rule Cloudflare ships on by default. I had not written that rule. I had not even chosen to be aggressive. The managed ruleset is on, and on Pro tier you cannot reconfigure it. You can only add exceptions on top.

The fix is counterintuitive

The instinct is to go loosen the managed ruleset so it stops eating crawlers. You cannot, at least not on Pro, and you would not want to anyway because that ruleset is also catching real junk. The right move is the opposite: add a higher-priority rule that tells Cloudflare to skip its protections for traffic it has already verified as a good bot.

The rule is one line:

  • Expression: (cf.client.bot)
  • Action: Skip
  • Skip: All managed rules, and All Super Bot Fight Mode rules
  • Logging: on

The load-bearing detail is cf.client.bot. That is not a User-Agent match. It is Cloudflare’s own cryptographically verified good-bot signal, built from reverse-DNS and ASN validation against the published crawler ranges. If some scraper forges Googlebot/2.1 in its User-Agent, it does not match cf.client.bot, because it cannot pass the reverse-DNS check back to Google’s infrastructure. So you get to wave through the real Google, Bing, and the rest without handing every spoofer a free pass.

If you have ever written a bot allow-list keyed on the User-Agent string, this is the upgrade. The UA is a self-declared label that anyone can type. The verified-bot signal is the provider doing the identity check for you. Trust the one that did the work.

Why a Skip rule and not a loosened ruleset

There is a second benefit that mattered for what I was doing next. I was about to flip Super Bot Fight Mode toward Block and turn the geo-block aggression up. A single high-priority verified-bot Skip rule covers both SBFM and the managed rules at once. That means before I make anything more aggressive, the crawlers I depend on are already carved out at the top of the stack. It shrinks the blast radius of every later tightening, because the one thing I cannot afford to break is already protected by a rule that runs first.

It also keeps the geo rule honest. I block everything outside the US and Canada on this site, since 100 percent of customers are US and Canada youth-sports leagues. A naive country block kills Googlebot crawling from a foreign datacenter, so the same (not cf.client.bot) clause has to live in that expression too:

(not ip.geoip.country in {"US" "CA"}) and (not cf.client.bot)

Without that second clause, the geo rule silently deindexes you the same way the managed ruleset did, just from a different direction. Verified-bot carve-out is not a one-time fix on one rule. It is a clause you owe every rule that can return a Block.

Verify against live events, not the config screen

Deploying the rule is not the finish line. The whole reason I caught this was that I refused to trust the summary, so I was not about to trust my own rule on faith either. I deployed, waited, and went back into the events.

This time the Google ranges were passing. A sampled address showed Action: Skip, and behind it two additional addresses from Google’s documented crawler ranges, all skipping. The events summary grew a brand-new “Skip: 156” line within about six minutes, and the 24-hour skip count climbed through the day from there: 940, then 1.95k, then 2.05k, and it was logging 8.44k verified-bot skips per 24 hours by end of day. That number climbing is the proof. A rule that exists in the config and a rule that is catching live traffic are two different claims, and only the second one means anything.

One more thing the events taught me: with overlapping defense layers, “why did this block” stops being obvious. A US probe I expected to pass came back 403, and it turned out to be the empty-User-Agent rule, not geo. When you stack managed rules plus SBFM plus a custom geo block plus a UA rule, any given block could be any of them. The Skip rule sitting at the top, logging on, is also your audit trail for confirming the good guys are getting through before the bad-guy rules ever see them.

The takeaway

“Turn on the CDN’s bot protection” is not a safe default for a content site. The managed rules ship on, and they will block the crawlers your business depends on, with zero error to tell you it happened. The damage is to your search ranking, and it shows up weeks later when it is hardest to diagnose.

Before you tighten anything, deploy a verified-bot Skip rule first, keyed on the provider’s cryptographic good-bot signal (cf.client.bot), not the spoofable User-Agent. Put the same (not cf.client.bot) carve-out in every rule that can return a Block, geo rules included. Then go read your live security events and watch real Google IPs land on Skip. If you have not seen the skip count climb on actual traffic, you have not verified the fix. You have only written it down.