---
title: "The Alert That Went Quiet Instead of Loud"
canonical: https://dxdev.com/blog/2026-08-29_alert-that-went-quiet-instead-of-loud/
datePublished: 2026-08-29
---
We built a small watchdog for one reason: tell us the moment a free trial ended and real charges started. Simple job. A balance check, a comparison against the last check, a message if something moved that shouldn't have. Then it missed two real charges in a row, for two completely different reasons, and neither failure looked like a failure. Both looked like a quiet night.

The first gap was almost funny once we found it. The alert had a setting meant to stop it from repeating the same warning every few minutes, a normal and sensible thing for any notification system to have. The problem was that two different kinds of alerts were sharing that one repeat-limiter. When the first alert fired and used up the quiet window, a second, unrelated real charge landed inside that same window and got treated as a repeat of something already reported. It wasn't a repeat. It was money moving that nobody heard about, because the tool assumed one silence meant "already told you" instead of checking what it was actually silencing.

The second gap was about clocks. The system we were watching resets and measures its day on one time zone. Our watchdog was checking "did this happen today" against a different one, the time zone of the machine running it. A charge that landed right around that boundary got filed under the wrong day, and the day-to-day comparison threw it out as if nothing had changed. That one mistake alone erased a real charge large enough to matter.

What ties both bugs together is the same shape: something that looked like a single, simple fact, a repeat window, a day, was actually standing in for two different things that needed to be told apart, and nothing in the setup forced that distinction. Neither bug crashed anything or produced an error a person would ever see. Each one just quietly agreed nothing had happened, which is the worst way an alert can fail. A watchdog that goes off too often gets noticed and fixed fast. A watchdog that goes silent at exactly the wrong moment is indistinguishable from a normal, uneventful day, until you go looking for the receipts.

The part that actually caught both problems wasn't a smarter read of the code. It was refusing to trust the code at all until it had been made to prove itself. We forced the exact condition that should trigger a real alert and checked, directly, whether a message actually showed up where a person would see it. Then we forced the opposite condition and confirmed nothing fired when nothing should have. Reasoning about the logic on paper had already happened once and it hadn't caught either bug. Making it actually go off did.

If you rely on any tool to tell you when something important changes, whether that's spend, an inventory count, or a security check, borrow the same test that caught both bugs here: force the exact condition that should trigger the alert, on purpose, and watch for a real notification landing where a person would see it, the same way forcing it exposed the shared repeat-key and the wrong-timezone comparison that reading the code never would have.