The Office Got Banned by Its Own IPFilter After 1,500 Hits

The office lost access to its own admin panel at 11:38 AM, and the block came from our own IP filter.

A local page-scanning macro had been running against our own sites. It crossed 1,500 fast-hits/day, and IPFilter did exactly what it was built to do: it blacklisted the source IP. That IP was the office’s. The tool we built to keep bad traffic out took out the whole office, including the internal admin panel we needed to fix it.

The rule fired exactly as designed

IPFilter watches request rate per IP and bans anything that crosses a threshold, because that is what scrapers and credential-stuffing bots look like from the server’s side. 1,500 hits in a day from one address is a reasonable bot signal.

What the rule had no concept of was whose IP it was looking at. There was no allowlist and no tag for “this address belongs to staff.” Nothing separated a scraper hammering a login form from someone running an internal QA scan against the sites we host. The filter had one dimension, request volume per IP, and once that number crossed the line the enforcement was identical regardless of source. The threshold was fine. The model was missing a dimension.

Diagnosing it from outside the perimeter

The report said we were being IP blocked by our own IP blocker and called it urgent. We first checked whether it was IPFilter or something upstream at Cloudflare, since we were mid-migration on a separate epic that same day and DNS and proxy issues were already on the radar. We ruled that out fast. The block was local, and traffic wasn’t even reaching the Cloudflare-fronted domains that hadn’t migrated yet.

Next we looked for the record of the ban. IPFilter’s state lives in the appcache database, not in a log file you tail. So the path was to SSH into the box, query appcache directly for the office IP’s ban entry, and confirm that the trigger reason and hit count matched the macro’s timing. They did.

We cleared the ban row. Then we added the office dev IP to a permanent allowlist entry in that same database, so the same macro run can’t re-trip it later.

To verify, we loaded the site from the office IP itself, not from a VPN or a staging box, to confirm the block was really gone and not just stale in some cache layer.

The fix that lost: turn off the macro

The easy fix on the table was to stop running the macro that hits 1,500 pages a day. That solves this incident but treats the symptom. The macro is legitimate internal tooling doing a legitimate job at a legitimate volume, and the defect is that it looks statistically identical to an attacker. If we throttled the macro instead of fixing the filter, the next piece of internal tooling that scales up traffic (a new monitoring script, a bulk migration check, anything hitting more than a handful of pages per minute) would walk into the same trap, and we’d be back here with a different urgent message.

So the allowlist entry, not a rate-limit tweak on the macro, is the fix. It encodes the missing dimension: a known-internal source overrides volume-based enforcement for that address. The threshold logic stays untouched for everyone else.

What generalizes

Any automated gate that acts on a single signal (request rate, error rate, failed-login count) will eventually score your own infrastructure as the threat, because your own infrastructure is the heaviest and most consistent user of the system it watches. Loosening the threshold doesn’t fix that. Giving the system a way to know what it already trusts, before it decides what to block, does. We didn’t have that carve-out for internal IPs going into this incident. We do now, and the next scanning tool we build against our own sites won’t be what locks us out of them.

dxdev LESSON · paste into your AI coding agent
LESSON: An automated filter with a single signal will eventually score your own infrastructure as the threat
SOURCE: dxdev.com/blog/firewall-self-lockout
WHAT HAPPENED: An internal page-scanning macro run against the company's own sites crossed 1,500 fast-hits per day, and the IP filter blacklisted the source address, which happened to be the office's own IP, locking staff out of the internal admin panel needed to fix it. The filter's logic was correct: 1,500 hits a day from one address is a reasonable bot signal, and the rule fired exactly as designed, but it had only one dimension, request volume per IP, with no allowlist and no way to distinguish a scraper hammering a login form from internal QA tooling hitting the same sites at similar volume. Diagnosis required ruling out an unrelated in-progress migration first, then finding that the filter's ban state lived in a database rather than a log file, querying it directly to confirm the trigger reason and hit count matched the macro's timing, clearing the ban, and verifying the fix from the actual office IP rather than from a VPN or staging box. The fix was an allowlist entry for the office IP directly in that database, not throttling the macro, because the macro was legitimate tooling and the next piece of internal tooling that scaled up traffic would trip the identical trap if the filter itself wasn't taught what it already trusted.
THE RULE: Any automated gate that acts on a single signal, request rate, error rate, failed-login count, will eventually flag your own infrastructure as the threat, because your own infrastructure is usually the heaviest, most consistent user of the system it's watching. The fix is giving the system a way to know what it already trusts before it decides what to block, not loosening the threshold for everyone.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any rate-based, volume-based, or frequency-based automated blocking rule with no allowlist or trust dimension distinguishing known-internal sources from external, unauthenticated ones.
  2. Any incident diagnosis for a blocking or lockout issue that didn't check the actual ban or block state directly, a database or state store, before assuming the cause, relying instead on log files or guesswork.
  3. Any fix for a legitimate internal tool tripping a defensive threshold that throttled the tool instead of teaching the filter to recognize the tool's known-trusted source.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.