The rate-limiter that banned our power users: 302 to 200 redirects look like an attack

A customer emailed to say they couldn’t reach the site. They attached a screenshot of an error page with one line of detail on it: Ref: 3405803818. That decimal number wasn’t an internal error code. It was their own IP address stored as a 32-bit integer, and our own application had blacklisted it because they browsed their admin pages too fast.

Not a firewall, not Cloudflare, not a real attacker. Our abuse detector decided one of our best customers was the threat, locked them out, and handed them a reference number that, once you decode it, points straight back at them.

The first wrong instinct

When a customer says “I can’t reach the site,” the reflex is to check the Windows firewall and the edge. That’s the wrong layer here, and chasing it cost us the first half hour. The block was in our own code the whole time.

The path in our code: the abuse-detection module calls a speed-checking stored procedure, and when an IP trips the speed flag the procedure sets a blacklisted flag on that IP’s database row and the request gets 302’d to our connection-error page. The customer never sees a firewall RST. They see our own error page, served by us, telling them they’re blocked. That page is where the Ref: decimal comes from: it renders the visitor’s IP in decimal form straight onto the screen.

The diagnostic lives in that one database row, not in the firewall rules.

Decode the screenshot first

Convert the Ref: value back to dotted form before you do anything else. That same day two more accounts hit the same wall: a team admin and an org-level account.

Once you have the dotted IP, you pull its database row. That row tells you whether it’s blacklisted, when, and by what set it. That row is the bug report; everything before it is guessing.

Our own redirects looked like a flood

The speed-flag counter increments on rapid same-host hits. The theory is sound: a scraper or a credential-stuffer hammers you with sub-second requests, you count them, you cut them off.

The problem is that our own admin pages generate exactly that traffic shape during normal use. A handful of legitimate admin actions emit paired same-host hits milliseconds apart, a 302 redirect immediately followed by the 200 it redirects to:

• the page-edit toggle
• custom-page navigation
• the login bounce
• a session-expired loop

To the counter, a 302 followed instantly by a 200 from the same host is indistinguishable from two abusive hits. One click becomes two counted requests, sub-second apart.

The worst offender was that last one. One account logged 251 hits to the session-expired login bounce URL in a single day. That’s not abuse. That’s a session-expiry loop bouncing a logged-out admin through the login page over and over, and every bounce was a free increment on the speed flag. The customer’s own broken session was feeding the machine that banned them.

The abuse was ours. The detector worked exactly as designed; it had just never been told what our normal traffic looked like.

The three-layer fix

This shipped as a hotfix, three layers, because no single one of them is enough on its own.

Layer A: teach the detector to recognize itself. In the abuse-detection module, check whether the Referer host matches the current host. If it does, tag the request as internal navigation, and the stored procedure skips the speed-flag increment for it. A 302 to 200 on the same site is navigation, not a flood.

Strip www. from both sides before comparing. The Referer might carry www. while the current host doesn’t, or vice versa, and a naive string match misses half the internal-nav traffic. The comparison is refStripped == curStripped, both normalized first. Get that wrong and you’ve built a same-host check that doesn’t catch same-host requests.

Layer B: make the ban decay, and only the heuristic ones. Auto-clear the blacklist flag after 7 days of no activity, but only if the stored procedure set it, not if a human did. A human-set ban is a decision, a heuristic-set ban is a guess, and the guess should expire on its own while the decision sticks.

If you don’t separate them, you either let real bad actors decay back in or you trap false-positive customers behind a ban that never lifts. Tagging who set the ban lets the decay apply only to the guesses.

Layer C: raise the threshold. With Layer A no longer feeding internal navigation into the counter, we raised the per-day ceiling from 500 to 1500. The two work together: Layer A clears out the false positives that were eating headroom, and 1500 is a realistic limit on what a busy admin actually generates.

Teach the detector what your own application’s traffic looks like first, then let it ban.

Related

• Your abuse detector is blacklisting your best customers: 302 then 200 looks like an attack: the companion incident narrative for the same event
• The 3% bot attack that took the site down: why your IP blacklist can’t see residential proxies: a different kind of detector failure, structural blindness rather than false positives
• Whack-a-mole is a real strategy: firewall blocks as a legitimate holding pattern: the other side of the same coin, when aggressive blocking is right
• It Looked Exactly Like a Bot Swarm. It Was SQL Server Parameter Sniffing.: another incident where the traffic pattern matched an attack but the cause was internal
• The Prod Box Was DDoSing Itself: An iCal Calendar Feature Looping Through the Public Edge: a self-inflicted traffic spike that looked like external abuse

dxdev LESSON - paste into your AI coding agent
LESSON: 302-to-200 redirect chains feed your own rate-limiter
SOURCE: dxdev.com/blog/rate-limiter-banned-power-users-302-200-redirects
WHAT HAPPENED: A per-IP speed-flag counter incremented on every same-host request, sub-second apart. Normal admin flows (redirect then final page, session-expired login bounce) produce exactly that shape. One user logged 251 hits in a day from a session-expiry loop alone. The detector worked as designed; it had never been told what normal traffic looked like. The fix was three layers: (1) skip the increment when the Referer host matches the current host (internal navigation, not a flood), (2) let heuristic-set bans decay after a period of inactivity while keeping human-set bans permanent, and (3) raise the threshold now that internal-nav noise was removed from the counter.
THE RULE: Teach your abuse detector what your own application traffic looks like before you let it make banning decisions.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Does the speed-flag counter increment on redirects from the same host (Referer host == current host, both www-normalized)?
  2. Are heuristic-set bans distinguishable from human-set bans, and does only the heuristic path have an expiry?
  3. Has the per-day ceiling been set with internal navigation noise excluded from the baseline measurement?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.