A customer in San Juan emailed support at 1:24 PM EDT to say the site was blocking them. Their email landed at the exact minute our Cloudflare logs show their IP starting to eat 403s, twenty-six of them across the next two minutes while their browser tried to load their own account’s pages. The geo-block we had shipped that morning allowed US and CA and blocked everything else. We never typed Puerto Rico into a deny-list. We didn’t have to. To Cloudflare, Puerto Rico is not the US.

The rule

We run a SaaS platform for customers in the US and Canada. One hundred percent of paying customers are in those two countries. The other ~144k foreign requests a day were scrapers, residential-proxy swarms, and bots with nothing to buy. Blocking it is structurally correct. The non-US/CA breakdown for a single day was Germany 74k, Singapore 15k, Brazil 14k, Vietnam 13k, France 12k, China 6k, Hong Kong 5k, India 5k. Zero funnel value, confirmed by the referrers (three US domains consistent with the customer base showed up in US/CA traffic; the German hits had no such referrers).

So the custom WAF rule was:

(not ip.geoip.country in {"US" "CA"}) and (not cf.client.bot)

Action: Block. The and (not cf.client.bot) clause is load-bearing for a different reason (it keeps Googlebot crawling from foreign datacenters out of the block, so you don’t deindex yourself), but that’s another post. The part that caused the problem was the set: {"US" "CA"}.

We verified it the morning we shipped it. check-host.net fires real HTTP requests from real country nodes, so we watched the result table come back: Brazil 403, Germany 403, Japan 403, UK 403. Canada 302, US 302. Exactly what we wanted. We shipped it and moved on to the next thing.

The block

A few hours later the San Juan ticket landed. The customer was specific: they were loading their organization’s pages and getting a block screen. So we traced their traffic.

Their IP was a Puerto Rico broadband address, geolocated to Puerto Rico. The Cloudflare events for that IP, starting at 17:24:26 UTC and running through 17:26:15, were a wall of 403s from our geo rule. Twenty-six blocks across that two-minute session, all of them their browser trying to pull assets off their own account’s site (their organization’s icons off the media subdomain). The minute they sent the email, 1:24 PM EDT, was 17:24 UTC, the same minute the blocks started. The logs proved it to the minute.

Then the part that makes you feel slow: of course. Cloudflare’s ip.geoip.country returns ISO 3166-1 alpha-2 codes. Under that standard, US territories have their own codes. They are not US. Puerto Rico is PR. The US Virgin Islands are VI. Guam is GU. American Samoa is AS. The Northern Mariana Islands are MP. Our allow-list said {"US" "CA"}, so a broadband subscriber in San Juan was, as far as the rule was concerned, exactly as foreign as the Singapore scraper we were trying to stop.

The country field is geographic, not political. It does not know or care that PR residents are US citizens, that the territory is on US carriers, that the traffic is your customer. ISO assigns Puerto Rico its own code, so Cloudflare treats it as its own country, so your US-only rule shuts out the whole island.

The fix

Add all five territory codes to the allow-list. We edited the rule live over the Cloudflare API:

(not ip.geoip.country in {"US" "CA" "PR" "VI" "GU" "AS" "MP"}) and (not cf.client.bot)

That went out as rule version 6, updated at 20:19:25Z. Five codes, not one, because if PR is its own country then so are the other four, and we were not going to ship this twice. VI, GU, AS, and MP have a fraction of PR’s population, but a Guam customer hitting the same wall next month is the same bug, and enumerating all five now costs nothing.

Any allow-list keyed on country codes has to enumerate the US territories explicitly. US is not a superset of PR VI GU AS MP, and neither the standard nor Cloudflare rolls them up for you. The US isn’t even the unusual case: France splits out its overseas departments (GF, MQ, RE, and more), and the UK has its own pile. If your business serves country X, check whether X has territory codes before you write in {"X"}, because the geo database will happily split them out and your customers there will silently eat 403s.

Verifying without lying about it

This is the part we want to be careful about, because it’s where it would have been easy to write a satisfying reply we couldn’t actually stand behind.

After the edit we pulled Puerto Rico’s traffic again. Zero PR blocks since the rule changed. In the roughly twelve minutes after the edit, PR traffic showed 216 requests served, 213 of them 200 and 3 of them 302. Island-wide, traffic was flowing and the block was gone.

What we could not see was the one customer. They hadn’t retried in that window. There was no event for their specific IP after the fix, because their browser wasn’t asking anymore. So we knew, with real evidence, that Puerto Rico in aggregate was being served. We did not know that their specific session was fixed. The only thing that would prove it is a request they hadn’t made yet.

So the reply said that. Something like “this should be working now, please confirm on your end,” not “you’re all set, it’s fixed.” A per-customer “fixed for you” is a claim about a thing we could not observe. “PR-wide traffic is flowing again, please confirm” is exactly what the data supported and no more. It’s tempting to round the aggregate success up into a personal one, because the customer wants to hear “fixed” and we wanted to say it. But we hadn’t watched their request succeed, so we didn’t get to say their request succeeded.

That care has a cost (the ticket stays open until they write back) and a payoff (you never tell a customer something is fixed and then have it not be, which is the one way to turn a few-hour outage into a credibility problem).

The rule design behind the geo-block

Every one of our customers is a US or Canadian organization, so the geo-block allows US and CA only. A day of Cloudflare traffic showed roughly 144,000 requests from outside the US and Canada. We checked the referrers before blocking, and they were what the business would expect: domestic organization sites. The foreign traffic was scrapers, residential-proxy swarms and bots. The rule also carries a verified-bot clause, cf.client.bot, so Googlebot and other real crawlers, which crawl from datacenters in many countries, are not blocked.

Takeaway

When confirming a fix, claim only what you observed: aggregate traffic recovering is not proof that one customer’s request succeeded.