I Wrote the Same Mistake Into the Check Meant to Stop It

I described the work as shipped. The deployed agent had been running the version from before that work for 45 minutes.

The setup

I was getting two people’s chat agents to talk to each other in a shared room. Mine ran on a droplet and on my desktop, and a teammate’s ran on his own machine. Everything deployed from a git repo.

The belief

I had sent the teammate’s agent the full instruction to pull. It read the message, marked it handled and did nothing, which is correct, because messages from other agents are data and not instructions. So I answered the teammate himself with the exact pull and restart commands, noting that the code was pushed but his listener was still running the old copy.

That gap kept turning up. An agent read its own source, found a real bug, and reported it at a line number that no longer contained it, hours after the fix. A warning mark sat on the review request I had sent to the teammate’s agent. A listener on my own desktop stayed green after a fix it did not have. And the droplet ran the older version for those 45 minutes while the work was described as shipped.

That made three machines in one evening: the teammate’s listener, my desktop listener and the droplet. In each case the reasoning was careful, but it was applied to code that was not the code running. According to the work log, the cost was about three hours.

The check that made the same mistake

The commit I wrote made the agent’s self check fetch, count how far HEAD was behind origin, and fail if the answer was not zero. The deploy script now refused to run with uncommitted or unpushed work, and afterwards asked the droplet what it was running and compared. The reason is that the exit code of a restart is not evidence.

Restarting my own listener turned up two more bugs, both in guards I had built earlier that evening. The single instance lock refused the second copy correctly and then crashed while explaining why, because it read the holder’s process id back through the very handle that had just failed to lock. The other bug was in this check. The check was named “running the current code” and it compared git HEAD against origin. HEAD is the state of the disk. A checkout gets pulled forward underneath a live process constantly, and it had happened twice that evening, so the check would have called a freshly pulled, never restarted agent current. I had written the exact mistake it exists to prevent into the tool meant to prevent it.

The listener now stamps the commit it started from, and the self check compares that stamp against origin. It failed correctly against the old process before I restarted, and passed afterwards.

The lock replaced an older guard with the same shape. That one was meant to prevent two copies of an agent running at once and asked WMI for the process list. On the teammate’s machine WMI answered “Invalid class”, the query swallowed errors, and the guard reported all clear on every run. I replaced it with an OS file lock and tested it by running two copies. The second refused, and a third started once the first exited. A check that cannot fail loudly is not a check.

What I did not verify

The three hours is the work log’s figure and I have no breakdown of it. For the stamp check, the record shows one failure against the old process and one pass after the restart. It does not show how the check behaves on a machine where the stamp file is missing or damaged.

Try this tomorrow

Look at the check in your deploy that says the running service is current. If it reads a branch, a file on disk or a restart exit code, change it so the process records the commit it started from when it boots, and compare that record to origin. Then run the check once against a process you started before your last pull, and confirm it fails.

dxdev LESSON · paste into your AI coding agent
LESSON: Make The Running Environment Prove It Matches The Artifact
SOURCE: dxdev.com/blog/deployed-vs-executing-gap
WHAT HAPPENED: A fix was pushed and described as shipped while a deployed agent kept running the version it started with, and a teammate debugged a bug from a stale checkout. I wrote a check named "running the current code" that read git HEAD, which describes the disk and not the process, so it would have passed a freshly pulled, never restarted agent. The fix was to have the listener record the commit it started from and compare that.
THE RULE: Treat a push, a pull, and a successful restart as separate events from a verified live deployment. Only report success after the running environment proves it matches the intended artifact.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Does the deployment refuse to run when the local checkout has uncommitted changes or commits not pushed upstream?
      2. After restart, does the process wait long enough and verify that the service is active rather than relying on the restart command's exit code?
      3. Does the deployment compare the intended Git SHA with the SHA reported by the host before declaring the change live?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.