Silence from a renewal job with no error line, no failed run, and 520 correct skips

A customer’s contact emailed support to ask why their site had gotten no renewal reminder since 2019, despite renewing several times in the years since. That was the whole complaint. There was no error on their end and nothing broken they could point to, just years of silence from a system that was supposed to warn them before their site lapsed.

The flag that never reset

The renewal-notice job runs on a schedule and checks every account against a handful of category windows: a month out, a week out, the day after expiry, and a long-lapsed follow-up. Each category has its own exclusion rule, meant to stop a customer getting the same warning twice in one cycle. The rule checked one thing: does this account’s “last notice sent” flag already equal this category’s name? If it does, skip it.

That is correct for exactly one cycle. Across cycles it was wrong, because nothing in the system ever reset the flag once the account renewed and a new cycle began. Once an account was flagged for, say, the “expiring in a month” category, it was excluded from that category forever, and the exclusion covered every later renewal as well as the one it was warned about. Renew five times and the account still reads as “already warned” from the first time, years earlier.

The account in the complaint had exactly that shape: flagged years earlier, renewed several times since, excluded the entire time. Every other check that could have blocked the notice for a legitimate reason came back clean. That covered a disabled trial, a test account and an opted-out group of accounts. One flag with no expiry was doing the opposite of what its own name implied.

Tracing it back, and the evidence that was already deleted

Finding the origin took longer than fixing it. The right design had existed once. An early version of this exact code checked whether the flag’s timestamp was recent before treating it as a suppression. That means don’t repeat a warning within the same cycle, but allow it again next cycle.

During a round of testing, that recency check was commented out. The commit message said something like “switched to a different approach, still testing.” The different approach was never finished and the recency check was never restored. Later, in an unrelated cleanup pass, the commented-out block was deleted outright, along with the field it referenced. That erased the last trace that a real fix had ever existed. From then on there was nothing to restore, only the bare equality check that had been running all along.

So for nearly seven years accounts slid into permanent exclusion and nothing looked like a bug from the outside. No failed job, no error line, and no support pattern anyone could name, because “never got a reminder” and “never needed one” look identical on a dashboard.

Measuring it before touching it

Before changing anything, I measured the scope against live data with bounded, read-only queries instead of trusting a code read. On the day the fix shipped, 27 accounts in the “month out” window and 1 in the “week out” window were newly eligible for a notice they had been blocked from. Roughly 520 accounts were correctly still inside a real single-cycle suppression and needed no change. That is no spike. It is a steady daily leak, and it never trips a threshold because there is no threshold for an email that didn’t go out.

The diagnosis and the planned fix then went to an independent second-opinion review with two questions: is the numeric scope right, and did the first pass miss anything? The core diagnosis held, but the planned fix was narrower than it needed to be. Two adjacent defects were sitting next to the one I was fixing:

• A failed or disabled email send still got recorded as sent. That creates the same kind of permanent skip through a different door.
• The eligible contact for a notice was resolved wrongly when the account’s stated owner isn’t its actual point of contact.

Both were real and both could ship as their own fixes. I tracked them separately instead of folding them into the same hotfix under time pressure. Bundling three differently shaped fixes into one urgent patch is how you introduce a fourth problem while fixing the first three.

Shipping it live, with no dry run

The fix restores what was commented out all those years ago. It compares the flag’s timestamp against each category’s own cycle window instead of checking bare equality forever. A flag from the current cycle still suppresses a duplicate, and a flag from any prior cycle no longer blocks the current one.

There was no separate backfill step, and that raised the bar for verifying before deploy. The moment the corrected query runs on its normal schedule, every account that had been excluded becomes eligible and gets a real email within minutes. That is the right outcome, but it means the fix ships once, live, to real inboxes. The only production check possible beforehand was the bounded read-only counts already run.

So I verified on a test account with the same stuck shape first. It confirmed the fix released that account without also releasing genuine same-cycle duplicates. Then I shipped it as a hotfix and re-ran the same count query against production afterward.

The query stays

That verification query is now a standing check instead of a one-off. A permanent flag posing as a one-cycle flag passes every test that doesn’t construct an account that renews more than once after being flagged, and nobody had built one in nearly seven years. A monitor didn’t catch this. A customer noticed silence where a reminder should have been, and someone decided that silence was worth the digging.

If you run reminder or billing-adjacent jobs, check whether any “already notified” flag resets when an account renews, and whether a test covers an account that renews more than once after being flagged. A missed notice leaves no failed job and no error line, so measure who is eligible against live data with bounded, read-only queries, and keep that query running.

dxdev LESSON · paste into your AI coding agent
LESSON: A Permanent Flag Only Means "Once" If Something Resets It
SOURCE: dxdev.com/blog/seven-year-old-code-bug-silently-blocks-customers
WHAT HAPPENED: A notification system suppressed a repeat notice to a customer by checking whether a flag on their record already equaled the notice category, with no time bound. A real recency check that would have expired the suppression after each renewal cycle existed in the original code, but was commented out during testing years earlier and never restored, then the dead code was fully deleted later, erasing the evidence it had ever existed. From that point on, any account that was ever flagged for a given notice category was permanently excluded from that category, forever, even after renewing many times since. Nobody filed a ticket because there was no error, no failed job, and no red line on a dashboard, an account that silently stopped getting a notice looks identical to an account that never needed one. It was found only because a customer's contact happened to ask support directly why no reminder had ever arrived.
THE RULE: A suppression or "don't repeat this" flag needs an expiry that matches the thing it's suppressing repeats of. If the flag is a bare value with no timestamp check, or the timestamp check that used to exist gets commented out "for testing" and nobody puts it back, the suppression silently becomes permanent, and nothing about that failure produces an error a monitor would ever catch. Before shipping a review-flagged fix like this, get a second, independent pass over the numeric scope and the adjacent code paths. In this case a second-opinion review found that the planned fix, while correct, did not cover two adjacent defects that would keep causing skipped or duplicate-recorded notices even after the main flag was fixed.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any "already sent" or "already handled" suppression flag that is a bare value (no timestamp) compared for equality, with nothing else in the codebase that resets or expires it on a schedule.
  2. Any commented-out or dead code near a suppression or dedup check, especially with a commit message suggesting it was disabled "for testing" and never explicitly re-enabled or removed with the original intent's replacement.
  3. Any fix for a customer-facing silent-skip bug that was verified only by the same person or process that diagnosed it, without an independent second pass checking whether the scope or the surrounding code paths were fully covered.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.