The IPs all looked clean. Every one of them had a fine reputation score, no entries on the usual blocklists, nothing a per-IP bad-rate check would ever flag. Meanwhile the origin box was pinned at 100% CPU and customers were timing out. The swarm was real. It was just hiding in the one place an IP reputation table never looks: the asset-mix.

If you run an app that anyone wants to scrape, you have probably reached for IP reputation as your first line of defense. It is the obvious metric. It is also blind to the attack that actually hurts you, which is a distributed residential-proxy swarm. Those run 20 to 30 IPs inside a single netblock (typically a /16 or /19 depending on the operator), each making a modest number of requests, every one of them clean in isolation. Rank by per-IP bad-rate and you see nothing. The swarm is sitting right under the threshold, spread thin on purpose.

Here is the method that actually catches it.

The three-signal triangle

App-layer scraper defense is a signal-triangle problem, not an IP-banning problem. You need three independent signals, and you do not call a swarm until all three point the same way.

1. Asset-mix. A real browser loading a real page pulls the HTML, then the CSS, then the images, the fonts, the JS. The ratio of page requests to asset requests sits in a normal band. A scraper does not care about your stylesheet. It wants the HTML, parses it, and moves on. So its asset-mix is abnormal: lots of page hits, almost no CSS or image hits. When the ratio of pages to assets inverts from what a browser produces, that is your first signal. This is the one that broke a case wide open during a swarm scan in May when nothing else had given a clear answer yet, and it is the one IP reputation cannot see at all.

2. User-Agent homogeneity. Real traffic is a mess of User-Agents. Different browsers, different versions, different OSes, mobile and desktop mixed together. A swarm driven off one tool tends toward UA homogeneity: the same string, or a tight cluster of strings, repeated across what should be unrelated visitors. Human traffic is diverse, so concentrated sameness is a tell.

3. Referer chain. A real user arrives somewhere with a plausible Referer history. They land on a page, click into another, and the Referer chain reflects that path through the site. Scrapers either send no Referer or send one that does not correspond to any real navigation path. A population of “visitors” with no plausible Referer chains, hitting deep pages cold, is the third leg.

One signal alone is noise. Asset-mix can look weird for a legit crawler you actually want. UA homogeneity happens for corporate networks behind one proxy. A missing Referer is normal for a bookmark. The discipline is the conjunction: you do not call a swarm until asset-mix is abnormal and UAs are homogeneous and the Referer chains are implausible, all in the same population. Three weak signals that agree beat one strong signal that is guessing.

Group by /16, not per-IP

This is the part that makes the triangle work, and it is the part people skip.

If you primary-group your logs by IP, a distributed swarm disappears. Each of its 20-to-30 addresses looks like a light, unremarkable visitor. The per-IP view is exactly the view the attacker is optimizing against.

Group by /16 netblock instead. Now those 20-to-30 sibling addresses collapse into one row, and that row shows the abnormal asset-mix, the homogeneous UAs, and the missing Referers all stacked together. The signal that was diluted across many IPs concentrates the moment you change the grouping key. Always primary-group by netblock and show the unique-IP-count per block as a column. A /16 with one or two unique IPs is a person. A /16 with 28 unique IPs all sharing a UA string and skipping every stylesheet is your swarm.

This generalizes beyond residential proxies. Distributed scrapers on cloud infrastructure spread the same way, across sibling addresses in one provider netblock. Per-IP cutting hides them; netblock grouping surfaces them.

The five-minute stop

Here is what this looks like as a playbook rather than a panic.

Late one night the origin server was at 100% CPU. Customers were timing out. The traffic was a German proxy-bot operator’s swarm. The earlier version of me would have started with per-IP analysis, found nothing damning, and burned an hour reaching for the wrong lever.

Instead, I ran the triangle. I pulled the logs, grouped by /16, and looked at the three signals per netblock. The asset-mix was the tell: almost no stylesheet or image hits in a block that had hit hundreds of pages. Once I saw that ratio, the rest followed. The abnormal asset-mix plus UA homogeneity within the block was the swarm, clear as day, the moment the grouping was right. From there it was one move: a Cloudflare WAF rate rule targeting the pattern at the netblock level, not a list of individual IPs to ban. The block went up, the swarm hit the edge instead of the origin, and CPU came back down. Active mitigation time was about five minutes.

The reason it was five minutes and not an hour was that I had already written the method down. This is the same discipline I used on an Awario scraper back in April, where I developed the fix and the playbook together in the same ticket, on parallel branches. A named playbook per bot class turns the next incident from a fire drill into a lookup. I am not banning IPs in a frenzy. I am matching a known pattern to a known mitigation. The codification is what collapses the time.

The counter-case: when it is not a swarm at all

The triangle is as valuable for what it rules out as for what it catches, and this is the part that keeps you from blocking paying customers.

A different production slowdown looked exactly like a swarm. CPU spiked. It correlated tightly with traffic. Every instinct said bot attack, and “it’s a bot attack” is a satisfying diagnosis that is easy to reach for. So I ran the triangle first. Asset-mix: normal. UA distribution: diverse, like real users. Referer chains: plausible. All three signals said human traffic. No swarm.

The real cause was a cached query plan in a high-frequency lookup path, catastrophically wrong for the actual input distribution. That incident is documented in full elsewhere. The short version: the fix was a one-liner, deployed in minutes, and response times dropped back to baseline. The point here is what the triangle did for me: it ruled out the swarm hypothesis before I reached for a WAF rule.

Before I let any mitigation rule go out now, I run one check: the netblock asset-mix has to be abnormal, the User-Agents have to be homogeneous, and the Referer chains have to be implausible, all three in the same population. Miss any one of the three and I hold off, because a false swarm call blocks real customers while the actual problem, a bad query plan or anything else, keeps running unfixed. That checklist is what kept a database problem from turning into a customer-blocking WAF rule.