Two Queued Failures Grouped as x2 in the Deferred Alert Flush

The 5:00 pm message said our data-collection job had failed twice overnight (exit code 2, about 93 seconds each run) and had not recovered since. The job’s last run, at 3:51 pm, had finished cleanly.

The job collects our infrastructure data every six hours, and an agent checked it for me. The scheduled task showed Ready, with a last result of 0. The data was about 75 minutes old, which is normal for a six-hour job. So the alert was wrong about “not recovered.” It was also wrong about “overnight.” The failures had happened during the day: from 6:41 am to 12:41 pm the connection to our web server kept timing out, and at 3:48 pm one run failed outright. A retry three minutes later worked.

The fix that missed

In the 3:48 run, the SSH connections timed out and the push then hit a DNS lookup failure, so it gave up after 93 seconds. The agent’s hypothesis was that the PC briefly had no network. It said so plainly as a hypothesis, since nobody had captured the network state at 3:48. It also found a bug in the alert itself and offered two fixes: stop treating a total loss of network as a failure worth an alert, and make the alert check the latest run before saying “not recovered.”

We went ahead with both. The push step got its own exit code 6 for network errors, and the scheduling script logged that as “push skipped” instead of a failure. The targeted tests passed (138). The full fast suite ran 15,276 tests and one failed, a known problem from earlier commits that day. We committed and pushed.

Then the 5:00 pm message got traced. The ledger showed it came from the queue where deferred alerts wait to be flushed together. Two failures were sitting in that queue: one at 534 seconds and one at 93 seconds. The flush grouped them as a single series, used the latest headline, and added “(x2)”. The 3:51 pm success never got written into the queue. The flush already knew how to drop a failure that had a matching “recovered” entry, but nothing ever wrote one.

The agent’s own verdict on our fix was that it “fixed the wrong path.” The network change would not have stopped that message anyway, because the server connections had also failed in that run. The fix did work. The agent ran the real script from a scratch copy with stand-in steps, and it exited 0 for an unreachable endpoint and 2 for a hard error. It just wasn’t the thing that sent the message. I had spent a full test run and a push on it before checking the path the message actually took.

The fix that matched

When any scheduled job goes from failing back to green, it now writes a “recovered” entry into the alert queue under the same name as the failure. The flush then drops the whole series. It applies to every scheduled job, not only this one, and it writes only when a failure is actually waiting in the queue, so a healthy job adds nothing.

Four new tests cover it:

• Fail, fail, recover leaves nothing to report.
• Fail with no recovery still reports “(x2)”.
• No write happens when nothing is queued.
• A recovered job produces no finding.

The full suite came back with 15,293 passed and 2 failed, and neither failure came from this change. One was the known problem. The other was a job registry test that passes when run alone, in a file where another session had uncommitted edits.

What has and hasn’t been seen

The next morning the agent reported that nothing had paged me overnight. The 6:41 pm run had timed out, but it ran before the retry fix and was held as a first failure, so it never alerted. The 12:41 am run was clean at 141 seconds and was logged as recovered.

A second change, a single retry after a 30-second pause when the server connection stalls, never had to fire. Neither it nor the “recovered” queue entry has been seen on a real stall. Both have only been tested in scratch runs, not on a real run.

The 5:00 pm message was an old failure reported after the job had already recovered at 3:51 pm, and it stayed in the queue because nothing recorded the recovery. The first real fail-then-recover cycle will show whether the “recovered” entry empties the queue.

dxdev LESSON · paste into your AI coding agent
LESSON: Trace An Alert To The Path That Actually Sent It Before Fixing Anything
SOURCE: dxdev.com/blog/alert-kept-firing-after-it-recovered
WHAT HAPPENED: The alert claimed two overnight failures with no recovery, while the scheduled task showed a last result of 0 and fresh data. The agent guessed the PC briefly lost network at 3:48 PM and proposed treating a total network loss as a skipped push, plus making the alert check the latest run. The team shipped both, ran the full suite and pushed. Only afterward did they trace the 5:00 PM message to the queue of deferred alerts, where two failures (534 and 93 seconds) were grouped as a series with "(x2)". The 3:51 PM success was never written into that queue, even though the flush already dropped failures that had a matching recovered entry. The network fix was real but would not have stopped that message, so the actual fix was to write a recovered entry whenever a job goes from failing to green with a failure waiting in the queue.
THE RULE: Before fixing an unexpected alert, trace the exact path that produced that message and reproduce it, rather than fixing the most plausible hypothesis about the underlying failure. A fix is only proven when the observed symptom is shown to go away through that path.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. For each alert you investigate, can you name the exact component and queue entry that emitted the message before any code change is made?
      2. Does every state transition from failing to healthy write a recovery record that the alert flush path can match against the earlier failure?
      3. Is each fix verified against the real symptom (the specific alert text or count), and is any part not yet seen on a real failure-then-recovery cycle explicitly marked as untested in production?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.