Your Deploy-Coherence Checker Is Only As Complete As Its Service List

I chased the weekly undeployed change warning for weeks before I found out it was a false positive. One service was absent from the checker’s own sidecar list.

That detail mattered more than the alert itself. A deploy-coherence check is supposed to answer a narrow question: did a change reach the environment it was meant to reach? When it says no, somebody has to assume there may be a real shipping gap. We did. The warning recurred, so we treated it as an operational problem, not background noise.

The warning looked like a deploy gap

The initial explanation was the obvious one. Something had changed, the expected deployment record did not line up, and the checker was flagging the difference. That is exactly the failure mode the tool exists to catch.

We also considered that the deployment state might be wrong, or that the checker’s comparison logic had regressed. Both were credible. A false negative lets a real gap through. A false positive teaches everyone to ignore a signal that should interrupt their day. Neither is harmless.

The cost was weeks, not minutes, and it was spent in the wrong place every time. Each recurrence sent me back into deployment records looking for a shipping gap that was never there, because the record I was checking was, by construction, always going to look consistent with itself. I was auditing the thing the checker compared against, never the checker’s own model of what existed to compare. That is weeks of investigation time against a symptom that a five-minute read of the sidecar list would have closed, if I had thought to distrust the tool instead of the system it was watching.

What finally worked was to stop asking whether the alert looked plausible and instead compare the checker’s inventory with the system it claimed to audit. I had spent the earlier rounds on the deployment records, which is where I would look for a real gap, and that is why it took me so long. The deployment state did not reveal an undeployed change. The scanner’s sidecar list revealed a missing service.

The checker had not found a change that failed to ship. It had evaluated an incomplete model of what needed to ship.

The gap was in the audit tool

That distinction changed the fix. I did not repair an application deployment. I added the omitted service to the scanner’s sidecar list and made the service inventory part of the thing we audit.

A sidecar list is easy to mistake for supporting configuration. It is not. In this kind of check, it is a boundary statement. It tells the scanner which services exist for the purpose of coherence. If one falls out of that list, the checker can continue running, continue producing a clear result, and still be wrong.

That is the dangerous version of configuration drift. Nothing crashes. There is no obvious exception. The tool returns an answer with enough shape to look authoritative.

We rejected the alternative of treating the recurring warning as a special case. Suppressing the alert would have hidden the symptom while leaving the inventory stale. We also rejected loosening the checker so it would tolerate the mismatch. A coherence check that tolerates unknown services is not checking coherence. It is checking a subset and presenting the result as the whole system.

Trust decays one false positive at a time

The technical change was small. The operational cost was not.

Every automated check spends trust when it interrupts someone. A correct alert earns that trust back because it points at a real condition. A weekly false undeployed change flag does the opposite. It trains the people receiving it to discount the next one, including the one that may describe an actual shipping failure.

The repair is not just to update the list once. The service list needs an audit path of its own. Whenever I add, remove, or split a service, the deploy checker has to prove that its sidecar inventory still matches the system boundary it is describing.

A deploy-coherence checker cannot be more complete than its service list. If that list is stale, the check may still run perfectly. It just will not be telling us the truth.

dxdev LESSON · paste into your AI coding agent
LESSON: A Coherence Checker Is Only As Complete As the Inventory It Audits Against
SOURCE: dxdev.com/blog/2026-08-22_deploy-checker-false-positive-sidecar-gap
WHAT HAPPENED: A recurring "undeployed change" alert was investigated for weeks by checking deployment records, the obvious place to look for a real shipping gap. The alert kept firing. The actual defect was in the audit tool itself: one service was missing from the checker's sidecar inventory, so the scanner was evaluating an incomplete model of what needed to ship and producing a confident, wrong answer every time. The fix was adding the missing service to the inventory and treating that inventory as something that itself needs an audit path whenever a service is added, removed, or split.
THE RULE: When a recurring automated check keeps flagging a condition that manual investigation cannot confirm, check the tool's own inventory or configuration before re-investigating the system it claims to audit; a coherence checker cannot be more complete than the list of things it believes exist, and it will return a clean-looking wrong answer instead of an error when that list is stale.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any monitoring, coherence, or drift checker whose service/resource inventory is a manually maintained list rather than something derived live from the deployment target.
  2. Any recurring alert that has fired more than twice with no confirmed root cause in the system it is nominally checking, where the checker's own configuration has not yet been audited as a candidate cause.
  3. Any process for adding a new service or sidecar that does not include a step confirming every coherence/audit tool's inventory was updated to include it.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.