AwarioBot Pushed CPU to 80 Percent on Our Legacy App Box

CPU on the production app’s box climbed to 80 percent and held for about twenty minutes before we dug into the session log. The slow-request log showed sort pages: the same URL pattern cycling through sort permutations, request after request from the same address block. User agent: AwarioBot.

AwarioBot is a brand-monitoring crawler. It isn’t malicious, just indifferent to the cost it imposes on a shared legacy application box. It was requesting the same logical page repeatedly with different sort parameters, and each request generated a full database round-trip.

Our first reflex was to pull the IP range and block it at the firewall. We asked Claude to surface the originating block. Then we checked the blocklist.

The range was already in there.

The seam that let it through

The production app’s bot-blocking has two layers. The first is a firewall-level IP blacklist, kept in a flat file and pushed to the box periodically. The second is a per-request filter file that runs on every request and handles dirty user agents, IP ranges flagged for review, and the connection-error redirect logic. The two layers are supposed to add up. They’re maintained separately and had never been audited against each other.

The second layer had an auto-clear rule. IP ranges showing clean activity over a rolling window got flagged as reviewed and removed from the active block queue. AwarioBot’s range had hit those pages roughly six months before this incident, with no 500s and no obvious abuse pattern. The rolling-window pass marked it reviewed and cleared it from the per-request queue. The firewall entry stayed. The per-request check moved on.

Claude’s initial read was right that AwarioBot was the load driver. Its suggestion was to tighten the IP-based block at the firewall level. That would have held until the auto-clear ran again, and then failed the same way. The seam doesn’t appear in any log line, which is why the IP fix looked correct.

Blocking it by name

The answer was simpler, and uglier. We added awariobot to the dirty user-agent array in the per-request filter and redirected matching requests to the existing connection error path, the same one other blocked bots hit. Thirty seconds of work.

On paper this is crude. A user-agent string is trivially spoofable. But AwarioBot is a commercial product with a published user-agent convention and no reason to disguise itself to scrape sort pages. A UA match is the right level of force here.

The bigger reason it was correct is that it works independently of the IP layer. The IP blacklist and the auto-review queue can be in any state they like. The UA check fires first in the per-request filter, before any IP logic runs, and the redirect exits the request immediately. Neither broken layer has to be fixed first.

CPU came back down to 35 percent within two minutes.

What the layers don’t know about each other

The IP blacklist and the per-request filter were designed to reinforce each other, and in the general case they do. But each has its own maintenance logic, its own update cadence, its own definition of “this range is now safe.” Neither knows what the other is doing.

A seam like this stays invisible until something crosses it. Ours showed up when a crawler we had blocked six months earlier reappeared from the same range after the auto-review queue had made the block meaningless. No log line says “this range is now covered by one layer only.” You find out when something breaks through.

We added the UA guard and moved on. The right time to audit the gap between those two layers is a quiet afternoon with the incident log as the spec, not while CPU is pegged. The dirty-UA array in the per-request filter is not elegant. It is the only thing that fired before either broken layer could matter, and that’s why the box is back to 35 percent.

If your site relies on more than one bot-blocking layer, find out whether anyone has ever checked them against each other. Here, an automatic review rule removed a range from one layer while the other kept it, and nothing recorded the gap. A block by name on a known commercial crawler worked in thirty seconds because it didn’t depend on either layer.

dxdev LESSON · paste into your AI coding agent
LESSON: Block by identifier when the IP layer has its own maintenance clock
SOURCE: dxdev.com/blog/block-the-bot-by-name
WHAT HAPPENED: A production app had two bot-blocking layers: a firewall IP blacklist and a per-request filter. The per-request filter had an auto-clear rule that removed IP ranges showing clean activity over a rolling window. A crawler that had been blocked at the IP level was quietly cleared from the per-request queue after a clean-activity window. When it returned, the IP blacklist entry was still present but the per-request check no longer fired, and the crawler drove CPU to 80 percent. The AI correctly identified the load driver but suggested tightening the IP block, which would have failed again at the next auto-clear cycle. The seam between two independently maintained layers was invisible in any single log line.
THE RULE: When two defense layers have independent maintenance clocks, add an identifier-based check that fires before both and exits the request without depending on either layer's state.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any place where a block decision is made: does it rely solely on IP matching or another value that a separate maintenance process can silently clear or expire?
  2. Any layered filtering setup: is there an audit trail or cross-reference that surfaces when one layer has cleared a range the other still considers blocked?
  3. Any auto-review or auto-clear rule applied to a security list: does clearing from one list also clear from all layers, or can layers fall out of sync silently?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.