The IPs all looked clean. Every one had a fine reputation score and no entries on the usual blocklists. Nothing in it would trip a per-IP bad-rate check. Meanwhile the origin box was pinned at 100% CPU and customers were timing out. The swarm was real. It was hiding in the one place an IP reputation table never looks: the asset-mix.

IP reputation is the obvious first metric for an app that anyone wants to scrape. It is also blind to the attack that does the damage, a distributed residential-proxy swarm. These run 20 to 30 IPs inside a single netblock (typically a /16 or /19 depending on the operator). Each IP makes a modest number of requests, and each is clean in isolation. Rank by per-IP bad-rate and you see nothing, because the swarm sits just under the threshold, spread thin on purpose.

Here is the method that catches it.

Three signals that have to agree

App-layer scraper defense is a signal-triangle problem. You need three independent signals, and you do not call a swarm until all three point the same way.

1. Asset-mix. A real browser loading a real page pulls the HTML, then the CSS, then the images, the fonts and the JS. The ratio of page requests to asset requests sits in a normal band. A scraper does not care about your stylesheet. It wants the HTML, parses it and moves on, so its asset-mix is abnormal: lots of page hits and almost no CSS or image hits. When the pages-to-assets ratio inverts from what a browser produces, that is your first signal. This is the one that broke a case wide open during a swarm scan in May, when nothing else had given a clear answer yet. IP reputation cannot see it at all.

2. User-Agent homogeneity. Real traffic is a mess of User-Agents: different browsers, versions and OSes, mobile and desktop mixed together. A swarm driven off one tool tends toward the same string, or a tight cluster of strings, repeated across what should be unrelated visitors. Human traffic is diverse. Concentrated sameness is a tell.

3. Referer chain. A real user lands on a page, clicks into another, and the Referer chain reflects that path through the site. Scrapers either send no Referer or send one that matches no real navigation path. A population of “visitors” with no plausible Referer chains, hitting deep pages cold, is the third leg.

One signal alone is noise. Asset-mix can look odd for a legitimate crawler you want. UA homogeneity shows up for corporate networks behind one proxy. A missing Referer is normal for a bookmark. So the rule is the conjunction: abnormal asset-mix, homogeneous UAs and implausible Referer chains, all in the same population. Three weak signals that agree beat one strong signal that is guessing.

Group by /16 instead of by IP

This is the part that makes the triangle work. It is also the part people skip.

If you primary-group your logs by IP, a distributed swarm disappears. Each of its 20 to 30 addresses looks like a light, unremarkable visitor. The per-IP view is exactly the view the attacker is optimizing against.

Group by /16 netblock and those sibling addresses collapse into one row. That row shows the abnormal asset-mix, the homogeneous UAs and the missing Referers stacked together. The signal that was diluted across many IPs concentrates the moment the grouping key changes. Always primary-group by netblock and show the unique-IP-count per block as a column. A /16 with one or two unique IPs is a person. A /16 with 28 unique IPs all sharing a UA string and skipping every stylesheet is your swarm.

This generalizes beyond residential proxies. Distributed scrapers on cloud infrastructure spread the same way, across sibling addresses in one provider netblock. Per-IP cutting hides them. Netblock grouping surfaces them.

The five-minute stop

Late one night the origin server hit the wall described above. The traffic turned out to be a German proxy-bot operator’s swarm. Starting with per-IP analysis would have found nothing damning and burned an hour reaching for the wrong lever.

Instead we ran the triangle. We pulled the logs, grouped by /16 and looked at the three signals per netblock. The asset-mix was the tell: almost no stylesheet or image hits in a block that had hit hundreds of pages. Once we saw that ratio, the rest followed. Abnormal asset-mix plus UA homogeneity within the block was the swarm, clear as day the moment the grouping was right.

From there it was one move: a Cloudflare WAF rate rule targeting the pattern at the netblock level, rather than a list of individual IPs to ban. The rule went up, the swarm hit the edge instead of the origin, and CPU came back down. Active mitigation time was about five minutes.

It took five minutes because we had already written the method down. This is the same discipline we used on an Awario scraper back in April, where we developed the fix and the playbook together in the same ticket, on parallel branches. A named playbook per bot class turns the next incident into a lookup. We match a known pattern to a known mitigation. The codification is what collapses the time.

When the triangle says it is not a swarm

The triangle is as valuable for what it rules out as for what it catches. That is the part that keeps you from blocking paying customers.

A different production slowdown looked exactly like a swarm. CPU spiked and it correlated tightly with traffic. “It’s a bot attack” is a satisfying diagnosis and an easy one to reach for, so we ran the triangle first. Asset-mix was normal. The UA distribution was diverse, like real users. Referer chains were plausible. All three signals said human traffic, so there was no swarm.

The cause was a cached query plan in a high-frequency lookup path, catastrophically wrong for the input distribution it saw. The fix was a one-liner, deployed in minutes, and response times dropped back to baseline. The triangle’s contribution was ruling out the swarm hypothesis before we reached for a WAF rule.

Before any mitigation rule goes out now, we run one check. The netblock asset-mix has to be abnormal, the User-Agents have to be homogeneous and the Referer chains have to be implausible, all three in the same population. Miss any one and we hold off. A false swarm call blocks real customers while a bad query plan, or whatever else is behind the slowdown, keeps running unfixed. That checklist kept a database problem from turning into a customer-blocking WAF rule.

Before any rule ships

Do not block traffic on one signal. Require abnormal asset-mix, homogeneous User-Agents and implausible Referer chains in the same population, grouped by netblock. Then check that the playbook is written down before the next incident.