The weekly undeployed change warning that sent us into the wrong records

We chased the weekly undeployed change warning for weeks before we found out it was a false positive. One service was absent from the checker’s own sidecar list.

A deploy-coherence check answers a narrow question: did a change reach the environment it was meant to reach? When it says no, somebody has to assume there may be a real shipping gap. We did. The warning recurred, so we treated it as an operational problem and stopped treating it as background noise.

Weeks in the deployment records

The first explanation was the obvious one. Something had changed, the expected deployment record did not line up, and the checker was flagging the difference. That is the failure the tool exists to catch.

We also considered that the deployment state might be wrong, or that the checker’s comparison logic had regressed. Both were credible. A false negative lets a real gap through. A false positive teaches everyone to ignore a signal that should interrupt their day.

So we went into the deployment records, and we went back into them every time the warning recurred. It cost weeks, and each round was spent in the wrong place. The record we were checking was always going to look consistent with itself. We were auditing the thing the checker compared against and never the checker’s own model of what existed to compare. A five-minute read of the sidecar list would have closed it, if we had distrusted the tool instead of the system it was watching.

What finally worked was to stop asking whether the alert looked plausible and compare the checker’s inventory with the system it claimed to audit. The deployment state showed no undeployed change. The scanner’s sidecar list showed a missing service.

The checker had not found a change that failed to ship. It had evaluated an incomplete model of what needed to ship.

The fix was in the audit tool

That changed what we repaired. We did not touch an application deployment. We added the omitted service to the scanner’s sidecar list and made the service inventory part of the thing we audit.

A sidecar list is easy to mistake for supporting configuration. In this kind of check it is a boundary statement. It tells the scanner which services exist for the purpose of coherence. If one falls out of the list, the checker keeps running, keeps producing a clear result, and is still wrong.

Nothing crashes and there is no exception. The tool returns an answer with enough shape to look authoritative.

We could have suppressed the recurring warning as a special case, but that would have hidden the symptom and left the inventory stale. We could have loosened the checker to tolerate the mismatch, but a coherence check that tolerates unknown services is checking a subset and presenting the result as the whole system.

What the false positives cost

The technical change was small. The operational cost was not.

Every automated check spends trust when it interrupts someone. A correct alert earns that trust back because it points at a real condition. A weekly false undeployed change flag trains the people receiving it to discount the next one, including one that may describe an actual shipping failure.

Updating the list once is not enough. The service list needs an audit path of its own. Whenever we add, remove or split a service, the deploy checker has to prove that its sidecar inventory still matches the system boundary it describes.

A deploy-coherence checker cannot be more complete than its service list. If that list is stale, the check can run perfectly and still not tell you the truth. When a deploy check keeps raising the same warning, test its inventory against the real system before you spend weeks in the deployment records.

dxdev LESSON · paste into your AI coding agent
LESSON: A Coherence Checker Is Only As Complete As the Inventory It Audits Against
SOURCE: dxdev.com/blog/deploy-checker-false-positive-sidecar-gap
WHAT HAPPENED: A recurring "undeployed change" alert was investigated for weeks by checking deployment records, the obvious place to look for a real shipping gap. The alert kept firing. The actual defect was in the audit tool itself: one service was missing from the checker's sidecar inventory, so the scanner was evaluating an incomplete model of what needed to ship and producing a confident, wrong answer every time. The fix was adding the missing service to the inventory and treating that inventory as something that itself needs an audit path whenever a service is added, removed, or split.
THE RULE: When a recurring automated check keeps flagging a condition that manual investigation cannot confirm, check the tool's own inventory or configuration before re-investigating the system it claims to audit; a coherence checker cannot be more complete than the list of things it believes exist, and it will return a clean-looking wrong answer instead of an error when that list is stale.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any monitoring, coherence, or drift checker whose service/resource inventory is a manually maintained list rather than something derived live from the deployment target.
  2. Any recurring alert that has fired more than twice with no confirmed root cause in the system it is nominally checking, where the checker's own configuration has not yet been audited as a candidate cause.
  3. Any process for adding a new service or sidecar that does not include a step confirming every coherence/audit tool's inventory was updated to include it.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.