1,282 Hostnames Audited and a Single Extension Caused the Real Trouble

At 2:54 p.m., the live production log showed the certificate auto-heal worker trying to process cf-heal-state.json as a command.

That file was the worker’s cooldown ledger, the record that stops it from re-triggering the same certificate repair over and over. We had written it as JSON because it was structured state. We had put it in the queue directory because that was where the worker was already writing files. The queue scanner treated every *.json file in that directory as work.

So the worker’s memory became work. No customer site broke, but we had built a loop that could make a healer chew on its own state every time the scanner ran.

It started with a site stuck without HTTPS

A migrated customer site had no HTTPS because its certificate had never completed. We unstuck that certificate by hand, then audited all 1,282 hostnames instead of assuming it was an isolated failure.

My first move was too broad. We re-triggered moved hostnames before there was a proper DNS gate in front of that action. The audit made the problem visible, and it also showed why an unbounded repair script was the wrong follow-up. A certificate re-trigger is not something to spray across every hostname that looks adjacent to the incident.

What we shipped had three layers. The domain dialog showed the true certificate state. Staff got a one-click re-trigger control for a known bad state. A background worker handled stuck certificates, so recovery no longer depended on a customer reporting a broken site.

The worker had three guards: a pointed-to-us gate, a cooldown, and a per-pass cap. The first kept it from acting on hostnames that did not point to us. The second kept it from retrying the same repair continuously. The third bounded how much work a single pass could do.

A manual button alone keeps recovery reactive. Re-triggering every moved hostname ignores the conditions that make the action safe. The worker was the right path only because its permissions were narrow enough to match the failure.

One glob was the command interface

The deployment used a file-backed queue. The worker scanned the queue directory with *.json, and any JSON file there was treated as a command payload. The filename extension was part of the protocol.

cf-heal-state.json lived in the same directory. To the code that wrote the cooldown ledger, it was a state file. To the queue scanner, it was a command. Each reading was reasonable on its own. Together they were a namespace collision.

At first the fault looked like it could be in the certificate flow: an API response, a bad hostname classification, or the cooldown logic itself. The worker had just gone live after a production certificate incident, so each of those was plausible. The production log showed the actual shape. The worker was not failing to remember a cooldown. It was reading the cooldown ledger through the wrong entry point and processing it as a bogus command.

That changed the fix. JSON parsing was never the issue. We were debugging an interface boundary that existed only because two unrelated files shared a directory and an extension.

State cannot look like work

The repair was small. The cooldown ledger stopped being a .json file in the queue namespace. The convention became: queue commands are JSON, cooldown ledgers use .txt.

A named exclusion for cf-heal-state.json would have patched this one instance and left the bad boundary in place. The queue would still mean both incoming commands and whatever support files happened to be stored there, and the next state file would have needed another exception.

A separate state directory would also work, and it is the cleaner choice when the worker has more state than a small cooldown ledger. Here, changing the ledger’s type was enough to make the contract explicit without touching the command scanner that was already working.

What mattered was one meaning per intake rule. *.json could mean “process this” again because the worker no longer wrote its own JSON state into that scope.

Remediation code needs a boundary first

The auto-heal worker exists because a stuck certificate should not wait for someone to notice that HTTPS is missing. But remediation code runs after something has already gone wrong, it has permission to make changes, and it often needs persistent memory to avoid doing the same thing twice.

That makes its storage choices part of the control plane. A cooldown ledger is not an implementation detail when it decides whether the worker acts. A queue directory is not just a folder when a glob turns filenames into commands.

We gave the worker a cooldown so it would not repeat repairs, then put the cooldown file where the worker looked for repairs. The live log caught the contradiction before it became a larger incident.

The final fix changed one file type. State and work now have different names and different meanings, and neither can pass for the other.

If you run automated repair, check where it keeps its own memory and confirm that memory sits apart from the work it picks up. Before it goes live, confirm every repair action is limited by a gate on which targets qualify, a cooldown, and a cap on work per pass.

dxdev LESSON · paste into your AI coding agent
LESSON: A file extension is part of a system's protocol; a state file sharing an extension with command files in the same directory can get processed as a command
SOURCE: dxdev.com/blog/automation-that-ate-its-own-state-file
WHAT HAPPENED: An auto-heal worker for stuck certificates kept a cooldown ledger, stored as a JSON file in the same directory the worker used as its command queue, to stop it from re-triggering the same repair repeatedly. The queue scanner treated every JSON file in that directory as a command payload, so the worker's own cooldown state file was picked up and processed as a bogus command, a namespace collision between two locally reasonable interpretations of the same file: state to one part of the code, work to another. The fix changed the cooldown ledger's file type away from JSON so it no longer matched the command scanner's pattern, restoring one meaning per file extension in that directory rather than adding a one-off exclusion for that specific filename, which would have left the underlying ambiguity in place for the next state file that needed to be written somewhere.
THE RULE: A file extension or naming pattern used to decide what a scanner or worker should do with a file is part of that system's protocol, not a cosmetic detail; any file written into the same namespace a scanner watches, including a system's own internal state, can be picked up and misinterpreted as work unless the meaning of that namespace stays reserved for exactly one purpose.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any file-backed queue or scanner that treats every file matching a glob pattern (like *.json) in a directory as work, with no separate namespace reserved for a worker's own internal state files.
      2. Any fix for a file misinterpreted as a command that adds a named exclusion for that one specific filename, rather than separating the namespace so the ambiguity cannot recur with the next new state file.
      3. Any remediation or auto-heal worker whose persistent memory (a cooldown, a lock, a checkpoint) is stored using the same format and location conventions as the inputs it treats as actionable commands.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.