A daily digest told its reader “3 things need a look”. When we read the three items, none of them needed anyone.

Seven defects in one message

The message had seven separate defects. The worst was a CI failure and its own later success for the same commit sitting in one message, one under NEEDS YOU and one under FYI. The full list:

  1. A red and its green for the same commit. Both entries now carry a match identity (repo, workflow, branch and commit). A red with a later matching green is demoted to FYI as a recovery line. It is never suppressed, so a build that flaps stays visible. The match is on that field, never on rendered text.
  2. A headline that counted something else. It counted escalated items, held items and items not addressed to the reader, but the held and not-addressed items render under WATCHING. It now counts only rendered NEEDS YOU rows. An empty count reads “nothing needed you overnight” and drops the “up from N” clause rather than trending against nothing.
  3. A log-shaper that mistook it for a log. Anything over eight lines got “Details logged: ” appended. We added an opt-out for messages already written for a person, rather than weaken a heuristic other callers rely on.
  4. A backslash Discord ate. In Markdown, \. is a valid escape, so Windows paths lost characters. Paths are now wrapped in a code span where the message is built, which fixes every caller.
  5. “1 item(s)”. It now uses a shared plural helper.
  6. “[check]” read like a dead link. It was the name of the failing job.
  7. Last night’s own delivery, recapped as news. Only entries starting with “digest:” were excluded from NEEDS YOU, so the prior night’s “morning-flush:” delivery came back as a fresh item. That is now a list of self-delivery prefixes, applied to the trend window too.

The commit went in at 2:46 PM with 128 tests passing across the changed modules and 5679 in the full fast subset. One failure in that subset was a spend-gate test that was already red on a clean checkout.

What the tests did not know

We did not stop at green. We rendered the real digest off the live ledger instead of the fixtures, and it still had “1 scheduled task(s) unhealthy” in it. A whole class had escaped, because every test we had written asserted the cases we already knew about.

We hunted for the class and found about 250 hardcoded (s), (es) and (ies) placeholders across the codebase. Fixing all of them would be churn, so the rule became scope by sink. A string is in scope only if it lands in a Discord DM, an SMS or a line rendered into a digest body. The other roughly 240 print to a terminal we read ourselves, or write a log entry, and stayed alone. The second commit at 4:08 PM fixed five files, each confirmed to reach a notification, with 87 tests passing across them.

That commit also listed seven more files as carrying the same defect that we had left for later. At 4:21 PM we went back and traced each one to its sink individually rather than replacing by pattern. Only four of the seven qualified. The other three were clean: one carried no such string at all and two only printed to a terminal. Two of the apparent hits in them were grep false positives on len(s), and one function was documented as addressed to a driver session, not to a person reading the digest. The full fast subset ended at 5678 passed and 0 failed, on our own run rather than a subagent’s report.

What it says now

The same morning renders like this:

dxdev, last 24h: nothing needed you overnight
WATCHING (1):
- <scheduled task> failed (rc=1, 446s)
FYI:
- CI recovered: <repo> ci@main, red for 1h 30m.

Three passes took from 2:46 PM to 4:21 PM, 95 minutes. The first one felt finished at 128 green tests. The stray “task(s)” line was the only thing that showed it was not, and it showed up only on a message a person would have read.

Takeaway

Passing tests only prove the cases someone already thought of. For any automated report people rely on, have someone read the real output built from live data, not just the test fixtures. When a defect turns up, ask for a fix scoped to where the output actually reaches a person, and ask that each claimed fix be traced to that point rather than applied by pattern.