Why Our Balance Check Filed a Real Charge Under the Wrong Day

Two real charges got past our watchdog in a row, and both nights looked like a quiet night. That was the whole problem.

We built the watchdog for one reason: tell us the moment a free trial ended and real charges started. Simple job. A balance check, a comparison against the last check, and a message if something moved that shouldn’t have. Each miss had a different cause, and neither one looked like a failure.

The warning that silenced the wrong charge

The alert had a setting meant to stop it from repeating the same warning every few minutes. That is a normal thing for any notification system to have. The trouble was that two different kinds of alerts were sharing that one repeat-limiter.

When the first alert fired, it used up the quiet window. Then a second, unrelated real charge landed inside that same window and got treated as a repeat of something already reported. It was not a repeat. It was money moving that nobody heard about, because the tool assumed one silence meant “already told you” without checking what it was actually silencing.

The charge filed under the wrong day

The second gap was about clocks. The system we were watching resets and measures its day on one time zone. Our watchdog was checking “did this happen today” against a different one, the time zone of the machine running it.

A charge that landed right around that boundary got filed under the wrong day. The day-to-day comparison threw it out as if nothing had changed. That one mistake erased a real charge large enough to matter.

Why neither one made a sound

Both bugs have the same shape. Something that looked like a single, simple fact, a repeat window or a day, was standing in for two different things that needed to be told apart. Nothing in the setup forced that distinction.

Neither bug crashed anything or produced an error a person would ever see. Each one agreed that nothing had happened, which is the worst way an alert can fail. A watchdog that goes off too often gets noticed and fixed fast. A watchdog that goes silent at exactly the wrong moment is indistinguishable from a normal, uneventful day, until you go looking for the receipts.

What found them

Reading the code carefully had already happened once, and it had not caught either bug. What caught them was refusing to trust the code until it proved itself. We forced the exact condition that should trigger a real alert and checked directly whether a message showed up where a person would see it. Then we forced the opposite condition and confirmed that nothing fired when nothing should have.

If you rely on any tool to tell you when something important changes, whether that is spend, an inventory count or a security check, do the same. Make the alert go off on purpose and watch for a real notification landing where a person would see it. In our case, that test exposed the shared repeat-limiter and the wrong-timezone comparison.

dxdev LESSON · paste into your AI coding agent
LESSON: An alert is only proven once it has actually gone off
SOURCE: dxdev.com/everyday-ai/alert-that-went-quiet-instead-of-loud
WHAT HAPPENED: A monitoring tool had two separate bugs that both suppressed a real alert without any error. Two different alert types shared one repeat-suppression key, so a real event inside the suppression window was silently dropped as if it were a duplicate. Separately, a "did this happen today" check compared a timestamp against the wrong time zone, so an event near the day boundary was filed under the wrong day and dropped by a day-over-day comparison. Both bugs were only confirmed fixed by forcing the alert condition on purpose and confirming a real notification landed, and separately confirming the no-alert case stayed quiet.
THE RULE: A value that looks like one simple fact, a repeat key, a day, a total, may actually be standing in for more than one thing that needs to be told apart; and any alert that watches for something important should be verified by forcing it to fire, not by reading the logic and trusting it.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any repeat-suppression, rate-limit, or deduplication key shared across more than one distinct alert or event type.
  2. Any "is this today" or date-range comparison that mixes a local clock with a fixed-timezone external system without an explicit conversion.
  3. Any alerting or monitoring code whose "does it actually alert" behavior has only ever been checked by reading the code, never by forcing the condition and watching a real notification land.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.