Theme
Bots and the edge
Telling real visitors from swarms, and what a CDN, WAF and geo rules will and will not do for you.
32 posts by DX, in two lanes: plain-language stories and the technical build log.
Everyday lane
Plain-language stories. No jargon. 3 posts.
Start here
Why Our Own Firewall Could Not Stop the Bot Swarm at 2am
A website attack became manageable only after we moved the first line of defense away from the machine being overwhelmed.
Stopping bot swarms with firewall rules
-
Why Our Own Firewall Could Not Stop the Bot Swarm at 2am
A website attack became manageable only after we moved the first line of defense away from the machine being overwhelmed.
-
4,000 requests from 19 sources and a screen that blocked nothing
After a rush of automated traffic, the durable fix was a simple map that showed us how to see the next problem clearly.
-
When the Bot Filter Blocked the Office's Own IP Address
An internal tool ran at a normal pace and tripped the same rule built to catch scrapers and credential-stuffing bots, locking staff out of their own admin panel.
Technical lane
The build log: the logs, the code, the fix. 29 posts.
Start here
Cloudflare Pro is a box of levers, not a managed service
Cloudflare Pro mitigated 0.9% of 1.98M requests. Pro is a toolkit of levers you configure yourself, not a managed service that protects you by default.
What a CDN really does for you
-
Cloudflare Pro is a box of levers, not a managed service
Cloudflare Pro mitigated 0.9% of 1.98M requests. Pro is a toolkit of levers you configure yourself, not a managed service that protects you by default.
-
JS Detections: the one Pro-tier Cloudflare lever that actually catches HTTP-replay proxies
Pro-tier bot rules never touched a residential proxy pushing 224k requests a day. JS Detections moved the fight to a capability it couldn't fake.
-
A Skip Rule on cf.client.bot and 8.44k Skips a Day
We clicked into a single sampled Cloudflare security event to see who our new bot protection was actually catching, and the top blocked source was a documented Googlebot address. That is Googlebot.
-
GPTBot vs OAI-SearchBot vs ChatGPT-User: blocking the wrong one deindexes you from AI search
Blocking GPTBot does not keep a site out of ChatGPT's results. OpenAI runs three different bots for three different jobs, and the per-bot toggles let you opt out of training without opting out of being found.
-
We Assumed We Were Letting AI Crawlers In. We'd Been Blocking Them for Months.
The alert fired on a rule we'd never named: 200 events in 24 hours through the verified-bot skip lane, and not one of them was the thing we'd come to document.
-
Letting AI Read Your Site Takes Four Layers
GPTBot was blocked twice, once by our edge and once by our robots.txt, while the answer bots got a 200.
-
"The Whole Island Can't Reach Your Site": US Territories Are Separate Countries to Cloudflare
A geo-block WAF rule that allowed US and CA silently locked out an entire US territory. How Cloudflare's ISO 3166-1 country codes work, the five territory codes to add, and a small lesson about what you can and can't claim when verifying a fix.
-
Why Cloudflare Full Mode Never Raised a 526 for Our Custom Domains
Cloudflare for SaaS can send SNI and Host header with different values, and IIS routes on Host only. On a Full zone the mismatch serves the wrong site silently.
-
Why Our Custom Cert Worked on www and Failed on the Root
Cloudflare Pro proxied every subdomain our SaaS threw at it and refused the apex.
-
Why support found nothing when they searched the Ref number from 104.23.x.x
The Ref number on the blocked page read 104.23.x.x. That's a Cloudflare edge address. The visitor it belonged to was on a residential VPN in Florida.
-
Why Our Allow Rule Allowed Nothing After the 520 Fix
Four edge rules took main-site downtime from about 39 minutes a day to zero within the hour. Later the same day we found out they were turning away real customer calendar feeds and search engines.
-
When the Firewall Log Stopped at v14
Cloudflare had 11 live custom firewall rules. The rule ID index still said six.
-
Two Weeks of Error Pages Behind One Customer's Short Address
A firewall rule written to stop a scanner also matched a real customer's own web address. It took an adversarial check to catch the outage nobody could see.
-
A 92% drop in error-minutes stalled at 670 failed requests a day
The blank error pages had been going on for weeks before anyone tied a number to them: 39 error-minutes a day, averaged over a 7-day window, out of 41 million requests.
-
Rate Limits and Bot Rules Cut Our Daily Error-Minutes by 91%
Our main site averaged 37.7 error-minutes a day before the rate-limit and bot-blocking rules went in at our CDN edge. Over the last five weeks it averaged 3.3, and yesterday it logged zero.
Stopping bot swarms with firewall rules
-
The 80% CPU spike that one user-agent string explained
Bot mitigation is a tiered decision. The same incident that called for a one-line UA block against a named crawler also needed a CIDR firewall rule against a distributed swarm. The wrong tool for each would have made things worse.
-
AwarioBot Pushed CPU to 80 Percent on Our Legacy App Box
AwarioBot slipped past an existing IP blacklist because the per-request filter had auto-cleared its range. Thirty seconds in the session handler fixed it.
-
The decorative database: when your blocklist table isn't actually on the request path
We had 1,186 blacklisted IP ranges the per-request stored proc never read. What the gap cost, and why a TCP-layer block beat the app-layer fix four to one.
-
One Windows Firewall Rule Took CPU From 73% to 11%
Top-IP-by-volume is the standard playbook for bot detection. Here is what happens when the attacker is 55+ IPs each hitting once, and why grouping by target URL beats grouping by source IP.
-
Why Per-IP Bad-Rate Found Nothing While the Origin Hit 100% CPU
IP reputation is blind to distributed residential-proxy swarms. Here is the three-signal method (asset-mix, UA homogeneity, Referer chain) grouped by netblock that stopped a live 100%-CPU incident in five minutes, plus the counter-case that turned out to be a database problem wearing a swarm's clothes.
-
Zero Images Across 612 Requests Gave the Scrape Away
A residential-proxy scrape got past every check we had. The fix was not another heuristic patch. We wrote a verdict schema before touching the detection code.
-
Operator, not IP: fingerprinting a 40,000-IP proxy swarm by its RIPE maintainer
Our production server had been pegged near 91% CPU for half an hour, and the offending traffic was spread across four dominant /16 netblocks with no two requests coming from the same IP.
-
The customer's own scraper IS the bot swarm you've been fighting
A customer opened a ticket asking us to allow-list their app's scraper so it could pull their own hosted site data.
-
CPU Sat at 100% While Our Log Watcher Reported Nothing Wrong
Five firewall rules stopped a proxy swarm. A log watcher covering one of three IIS sites kept the attack invisible.
-
Whack-a-mole is a real strategy: firewall blocks as a legitimate holding pattern
IP-range blocking has a bad reputation. This is a defense of it as deliberate triage: when to reach for it, how to scope it to avoid collateral damage, and how to make sure each block teaches your stack instead of just buying time.
-
The rate-limiter that banned our power users: 302 to 200 redirects look like an attack
How our own rate-limiter banned a power user because 302-to-200 redirect chains look identical to a flood. A three-layer fix that taught the detector what normal traffic looks like.
-
Two hours of notes after the ticket closed on April 23rd
A ticket closed on April 23rd with five firewall rules. The durable win was the two hours we spent the next morning writing down how to investigate the next swarm so we would not have to rediscover the frame from scratch.
-
The Office Got Banned by Its Own IPFilter After 1,500 Hits
The office lost access to its own admin panel at 11:38 AM. The block came from our own IP filter, and it hit our own staff.
-
Cloudflare Rulesets API and the 2am bot swarm
Near-daily bot swarms were crawling or downing the site, and every rule we wrote landed after the swarm had already moved on. The fix was admitting the firewall lived on the wrong machine.
Hitting one of these walls in your own codebase or your own machine? Start an assessment or talk to us.