Topic Guide
Bot Defense and Scraper Mitigation
Every post in this hub started with the same page load: a graph that shouldn't be climbing, a CPU number that shouldn't be that high, or a support ticket from a real customer who just got firewall-blocked by mistake. Bot defense on a live site is rarely one clean fix. It's detection first, telling a genuine swarm apart from a slow query pretending to be one, or your own redirect chain tripping your own abuse filter. Then a proportional response: a name-based block for a named crawler, a CIDR firewall rule for a distributed swarm, a WAF lever for a residential proxy pool that IP reputation can't see.
The posts below are grouped by what they're actually about, not by publish date. Some are pure detection method. Some are the embarrassing kind, where the attacker was us. Some are Cloudflare and WAF configuration notes, because Cloudflare Pro is a box of levers you have to pull yourself, not a managed service that protects you by default. A few are about the moment the logs themselves turned out to be lying.
If you run a real site behind a real edge, the throughline worth taking is this: the fix that survives isn't the one that stops the traffic, it's the one that teaches your monitoring something, so the next swarm starts from a playbook instead of from zero.
21 posts in this guide, by DX
Start here
Bot Swarm Detection: The Three-Signal Triangle That Catches What IP Reputation Misses
IP reputation is blind to distributed residential-proxy swarms. Here is the three-signal method (asset-mix, UA homogeneity, Referer chain) grouped by netblock that stopped a live 100%-CPU incident in five minutes, plus the counter-case that turned out to be a database problem wearing a swarm's clothes.
Telling a real swarm from a false alarm
-
The bot-swarm that broke the playbook: URL fan-out beats top-IP-by-volume
Top-IP-by-volume is the standard playbook for bot detection. Here is what happens when the attacker is 55+ IPs each hitting once, and why grouping by target URL beats grouping by source IP.
-
JS Detections: the one Pro-tier Cloudflare lever that actually catches HTTP-replay proxies
Pro-tier bot rules never touched a residential proxy pushing 224k requests a day. JS Detections moved the fight to a capability it couldn't fake.
-
It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.
Traffic-correlated CPU spikes can mean a scraper swarm or a cached bad execution plan. Here is how to tell them apart and what to do when the answer is the plan.
-
The decorative database: when your blocklist table isn't actually on the request path
I had 1,186 blacklisted IP ranges the per-request stored proc never read. What the gap cost, and why a TCP-layer block beat the app-layer fix four to one.
When the attacker was us
-
The attacker was our own redirect logic
The abuse filter caught an attacker. It was us. Our own redirect chain was tripping the speed flag on real admin accounts.
-
The call was coming from inside the house
A login abuse filter blacklisted a power user for clicking too fast; the IIS logs proved the fast clicks were our own redirect chain, not an attack.
-
The rate-limiter that banned our power users: 302 to 200 redirects look like an attack
How our own rate-limiter banned a power user because 302-to-200 redirect chains look identical to a flood. A three-layer fix that taught the detector what normal traffic looks like.
-
The SSL bug was easy. The bot traffic made it a day-long incident.
A certificate-renewal failure became a six-release incident as bot traffic and slow feedback compounded the original problem. A retrospective on why observability belongs in an incident response loop.
Cloudflare and WAF configuration, the levers and their traps
-
Cloudflare Pro is a box of levers, not a managed service
Cloudflare Pro mitigated 0.9% of 1.98M requests. Pro is a toolkit of levers you configure yourself, not a managed service that protects you by default.
-
Geo-blocking everywhere except the US and Canada, and the one clause that saves your SEO
How to write a Cloudflare geo-block WAF rule that covers your real customer base without silently deindexing you, plus the US-territory gotcha that bites immediately after.
-
"The Whole Island Can't Reach Your Site": US Territories Are Separate Countries to Cloudflare
A geo-block WAF rule that allowed US and CA silently locked out an entire US territory. How Cloudflare's ISO 3166-1 country codes work, the five territory codes to add, and a small lesson about what you can and can't claim when verifying a fix.
-
Your CDN's default bot protection is blocking Googlebot and quietly bleeding your SEO
I clicked into a single sampled Cloudflare security event to see who my new bot protection was actually catching, and the top blocked source was a documented Googlebot address. That is Googlebot.
-
GPTBot vs OAI-SearchBot vs ChatGPT-User: blocking the wrong one deindexes you from AI search
"Does blocking GPTBot keep me out of ChatGPT's results?" The answer is no, and the reason is the whole point of this post: OpenAI runs three different bots for three different jobs, and the per-bot...
Fast fixes under fire
-
The fastest fix was blocking the bot by name
AwarioBot slipped past an existing IP blacklist because the per-request filter had auto-cleared its range. Thirty seconds in the session handler fixed it.
-
The UA blocklist is the right-sized fix for an identified crawler (and the wrong one for a swarm)
Bot mitigation is a tiered decision. The same incident that called for a one-line UA block against a named crawler also needed a CIDR firewall rule against a distributed swarm. The wrong tool for each would have made things worse.
-
Whack-a-mole is a real strategy: firewall blocks as a legitimate holding pattern
IP-range blocking has a bad reputation. This is a defense of it as deliberate triage: when to reach for it, how to scope it to avoid collateral damage, and how to make sure each block teaches your stack instead of just buying time.
When your own logs lie to you
-
Your IIS Logs Start Lying the Moment Cloudflare Goes Live
Orange-cloud a legacy IIS app and every IP your origin sees becomes a Cloudflare edge IP, with no errors anywhere. How to find and fix the lying logs.
-
Five firewall rules beat the proxy swarm, but the real bug was my blind spot
Five firewall rules stopped a proxy swarm. The real bug was a log watcher covering one of three IIS sites, which kept the attack invisible.
The method, not just the patch
-
The method outlasts the patch
A ticket closed on April 23rd with five firewall rules. The durable win was the two hours I spent the next morning writing down how to investigate the next swarm so I would not have to rediscover the frame from scratch.
-
The bot swarms moved faster than my firewall rules
Near-daily bot swarms were crawling or downing the site, and every rule I wrote landed after the swarm had already moved on. The fix wasn't a better rule. It was admitting the firewall lived on the wrong machine.
Hitting one of these walls in your own codebase or your own machine? Talk it through with us, or read the rest of the Build Log.