---
title: "Our Alerts Watched for Failed Builds While the Plan Quietly Stopped Running"
canonical: https://dxdev.com/blog/external-tool-state-unmonitored/
datePublished: 2026-08-30
---
Our CI never failed. What I got instead was a teammate asking twice, five days apart, whether his preview build had actually gone out.

It had not. Every push to that branch hit our CI, ran clean, and reported success. The plan itself had been switched to disabled sometime in the prior week. Our CI system does not reject a build against a disabled plan. It accepts the trigger, marks the job as skipped, and returns green. From the pusher's side there is no difference between "deployed" and "silently discarded." Same webhook response, same commit status check, same nothing in the CI tab that would make you look twice.

## Working backward from a stale preview

We started from the symptom, not the log. The teammate said the preview site looked stale. We diffed the commit hash the site was serving against `git log` on the branch, and they were five days apart, right at the point his last real deploy had landed. Then we went looking for the failure and there wasn't one. No failed run, no error email, no red X anywhere in the build history, because there was no build history for those five days at all. The plan just stopped showing up in the list of things that ran.

## The cache theory that cost forty minutes

Our first move, before we went into the CI system's UI, was to assume a caching problem on the preview host. Preview environments in this stack sit behind a CDN layer that we've had misbehave before, serving a stale build after a real deploy because of a cache key that didn't bust cleanly. So we bounced the CDN cache for that host, hit the URL again, and watched the same five-day-old commit hash come back. We lost maybe forty minutes chasing a cache invalidation that was never going to fix anything, because the new build had never been produced. The deploy artifact for the last five days did not exist on the target. The cache had nothing to invalidate.

That wrong turn pushed us to stop trusting "the pipeline says success" and start comparing what the pipeline reported against what was actually on disk on the target host. Once we did that, the gap was obvious in about two minutes: five days of green checkmarks with zero corresponding artifacts.

## The audit log had nothing

We checked the CI system's audit log for who or what disabled the plan, and there was nothing to check. No entry, no timestamp, no actor. Either it predates what the instance retains, or the disable happened through a path that doesn't log to that table at all. We re-enabled the plan, kicked a manual build, confirmed the site was now serving his actual latest commit, and closed that part out. The cause is gone. It isn't redacted or buried in a log we didn't look at hard enough. It is unrecoverable.

## What we filed

The immediate fix was one click: re-enable the plan. The longer fix is a ticket we filed to track the monitoring gap. Config drifts and people fat-finger toggles, so a disabled plan is going to keep happening. What let it go unnoticed for five days is the monitoring, which watches for jobs failing and has no opinion on a job that stops running entirely. A disabled plan doesn't fail. It stops existing as a thing that could fail, and every downstream check that says "no failures reported" reads that as healthy.

We already had an alert for when the build fails. What we filed is a heartbeat check: has plan X produced an artifact with a timestamp newer than N hours, independent of what the CI system's own status API claims. If the last successful artifact is stale, page someone, regardless of whether the plan even ran. That's the only version of this check that would have caught it, because from the CI system's own point of view, nothing had gone wrong. Nothing had gone at all.

A green build status does not prove anything was deployed. In this case a disabled CI plan produced five days of green checkmarks with no builds behind them, and the only check that can see that is one that looks for a fresh artifact on the target within a set number of hours.