The site was slow, the CPU on the IIS box was sitting around 80%, and the loudest thing in the logs was a single named crawler. AwarioBot, a brand-monitoring service, was walking listing pages in every sort permutation it could construct, one query-string variation after another, racking up roughly 1,600 hits from a Hetzner IP. I killed it with one line:

"awariobot":1

That went into the UA blocklist dictionary in the request filter. No firewall rule, no IP blacklist, no whois lookup, no expensive backend lookup. One key in a dict, and every subsequent request carrying that user-agent got 302’d straight to the error page before it touched the expensive page-render path. CPU came back down. That was the whole fix for the visible symptom.

The reason this is worth writing up isn’t that user-agent blocking is clever. It’s the opposite. Bot mitigation is a tiered decision. The same incident that surfaced AwarioBot also had me in a firewall-CIDR fight against a completely different attacker on the same box, a distributed scraping swarm that no user-agent rule could have touched. Picking the wrong tier for the shape of the attacker in front of you either misses the attack entirely or blocks real paying customers. The UA blocklist was exactly right for AwarioBot and would have been useless against the swarm.

Why the UA block was the correct tier here

AwarioBot has three properties that make it a textbook UA-blocklist target:

  1. It is identified. It announces itself in the user-agent string. It is a real, named, commercial crawler with a fixed UA. It is not trying to hide.
  2. The UA is the stable identifier, and the block is source-agnostic. A UA rule fires on the header, not the address, so it does not matter whether the crawler hits you from one IP or rotates through a hosting provider’s whole block. AwarioBot here was concentrated on a single Hetzner IP, but I never had to care, because the rule keys on the one thing the crawler keeps constant: its name.
  3. The cost of a false positive is near zero. Nobody legitimate is browsing your registration site with awariobot in their user-agent. There is no collateral damage. You are not going to 302 a paying customer making a legitimate request.

When all three of those hold, the user-agent string is the cheapest stable identifier you have. This is the same pattern already protecting the box against a handful of other named crawlers; AwarioBot just joined the list. The request filter checks the incoming UA against the dict and bounces matches before any real work happens.

That last point is the performance argument for doing the block at the UA layer rather than the IP-filter layer: you reject the request early, before the page does any database work to build the response.

The companion bug: the IP that should already have been blocked

Here is the part that makes the “match the tier to the attacker” point concrete rather than theoretical. The AwarioBot IP was sitting inside a range already flagged as bad in the IP reputation data. In principle it should have been caught one layer down, by the IP filter, and never reached the point where I had to add a UA rule at all.

It got through because of a decay bug. The IP filter had a secondary tier, added in an earlier pass, that auto-cleared its per-IP blacklist entries after 7 days. An IP it had correctly blocked got silently un-blocked a week later, and the crawler sailed back in. The real second half of this fix was removing that 7-day auto-decay so a deliberately-blocked IP stays blocked. The UA rule stopped the bleeding. The decay removal closed the door the IP had walked through. (There’s a deeper hole I’m not even getting into here: that filter only ever checks the per-IP blacklist flag and never consults the broader IP range data at all, so range membership was decorative intelligence the request path didn’t enforce. That’s its own post.)

A blocklist that auto-expires its own entries is not a blocklist, it’s a suggestion with a timer. If you flag something as bad, it stays bad until a human un-flags it. Decay is a feature you add for organic noise, not for confirmed abusers.

The other attacker the same night, and why the same fix would have failed

Now contrast the AwarioBot job with the thing the UA blocklist could not have touched.

The other half of that same incident was a distributed scraping swarm. The signature was completely different: 55-plus unique IPs, each hitting a single page only one or two times, almost all of them inside the same hosting ASN (Alibaba Cloud). Rank that traffic by hits-per-IP, the way a classic “block the loudest IP” playbook tells you to, and you find nothing. No single IP is loud. The volume is spread thin across dozens of sources precisely so that per-IP analysis sees noise.

A UA block would have done nothing against this. Swarms either rotate UA strings or, more often, just send a generic real-browser UA string. There is no awariobot:1 line you can write because there is no stable identifier in the header to key on.

The right tier for that attacker was the firewall, blocking by CIDR. One Windows Firewall rule against the hosting provider’s published range took the box from roughly 73% CPU and ~1000 connections down to 11% CPU and ~128 connections in seconds. That works because a TCP-dropped packet costs the IIS process zero CPU. The connection never gets to the application. The UA block still processes the request far enough to issue the 302, which is cheap but not free. Under a real flood you want the packet dropped at the firewall, not bounced at the app.

And the firewall tier has its own trap, which is why it is the heavier tool. Within hours of blocking that range, the swarm rotated to a neighboring block that wasn’t in our data at all. I assumed at the time it was more of the same provider’s space. It wasn’t. Checking the public allocation registry (RDAP) showed the new block belonged to a completely different cloud provider. So the rotation wasn’t within one provider’s range, it was across providers, and my mental model of “block the attacker’s ASN” was wrong before I finished typing the rule. That is the real lesson of the firewall tier: your blocklist is built from the traffic you happened to see, and the attacker is not confined to it. Blocking by range means consulting the actual public allocations, not your own stale subset. And you have to be careful where you swing: some blocks that look like clean hosting-provider space sit right next to carrier mobile ranges. A lazy wide block takes out real customers on their phones. That is the exact false-positive cost that does not exist when you block awariobot by UA. The firewall tier is powerful and dangerous, and you only reach for it when the attacker’s shape forces you to.

The decision

Before you write any rule, answer one question: what is the stable identifier for this attacker?

If it’s the user-agent, block the UA. If it’s the netblock, block the CIDR. If it’s a single IP, block the IP. The failure mode in both directions is using the wrong tier. Reach for the firewall against AwarioBot and you start blocking Hetzner CIDRs and risking real users on shared hosting, when one dict key would have ended it. Reach for the UA blocklist against the swarm and you spend an afternoon writing rules that match nothing because the attacker has no stable string to key on.

The same night this incident closed, I wrote down the triage check I now run first: name the attacker’s stable identifier (a UA string, a single IP, or a CIDR), then go confirm the filter that’s supposed to key on it is actually enforcing, the way the 7-day decay bug proved a working-looking IP filter can quietly stop doing its job. If the identifier and the enforcement don’t line up, fix that before writing a new rule.