Slot 022 lost its session manifest and nothing told us

Slot 022’s session manifest was gone when we went to log a DNS cutover check. The check itself was routine: a customer domain needed verification of its CNAME, apex A, cert, MX and our own domain record, which takes a few minutes. Another session’s close routine had already run and deleted our session’s active manifest. That file records what a session is doing and lets a human or another agent find it later.

We filed a ticket for the deletion itself, restored the manifest at slot 022, and moved on, but only after we noticed. We don’t know how long our session would have run with no manifest if we hadn’t happened to go log something right then. Every tool that resolves a session by reading that file would have started failing in ways that look like unrelated bugs, not like a deleted file. That is a worse failure than an error message, because nothing would have told us to look. The near-miss was a predictable failure mode of running many concurrent Claude Code sessions against one shared git-backed repo with no lock on manifest deletion.

How the shared repo allows this

Session state lives in a single git repo. Sessions write manifests into it, one file per session id. A pre-close routine runs when a session ends: check for uncommitted work, check the task board, and, apparently, clean up. That cleanup step didn’t scope its deletes tightly enough. A /close, or a cleanup pass adjacent to it, treated another live session’s manifest as safe to remove. Presumably it looked orphaned by whatever heuristic applied, maybe an age check, maybe a status field, and it wasn’t.

The same failure class shows up in desk_authority.py, which came up in the same day’s logs. That module manages a single shared “desk lease” so that only one Claude Code session at a time can move browser windows around our screen. Two sessions grabbing the desk to surface a decision page at once would fight over placement, so there’s a lease:

STALE_WINDOW_SEC = 30 * 60

def is_stale(lease: dict) -> bool:
    age = _age_seconds(lease.get("last_touch", ""))
    return age is None or age > STALE_WINDOW_SEC

def lease_state(sid: str) -> dict:
    ...
    if is_stale(lease):
        return {"status": "stale", "lease": lease, "age_seconds": age}
    return {"status": "held", "lease": lease, "age_seconds": age}

A lease is held if another session touched it within the last 30 minutes, stale otherwise, and owned if it’s yours. _write_json writes atomically, tmp file then os.replace, so two sessions can’t both believe they hold the lease from a half-written file. That is the right instinct: never let concurrent writers race on the same piece of shared, meaningful state. A queue of pending requests sits behind it, so a session that can’t take the lease immediately can ask and wait rather than force it.

The manifest deletion had no equivalent guard. Nothing staled it, nothing queued a request to remove it, and nothing atomically checked whether it was actually orphaned before unlinking it. It just went.

Git was the first thing we used. The repo is versioned, so restoring slot 022 was a git checkout on that path once we noticed. It worked. But it only works if you notice, and that is the limit. Our session kept running with no manifest, and git being able to restore the file afterward does nothing for the stretch of time, however long it was, where nothing knew the file was missing. The damage was silent state corruption, and no data was lost. We’re keeping git as the last-resort backstop.

Lock the manifest directory like the desk. Reuse the desk_authority.py lease pattern. Before any close routine deletes a manifest, it has to claim a lease on that specific sid’s file, which fails if the file was touched inside the stale window. This is the fix we’d converge on, since the mechanism already exists and is proven. It’s a matter of pointing it at manifests instead of only at desk placement.

Never delete, only archive. Have close routines move manifests to a closed/ subdirectory instead of deleting them. This is simpler and safer by construction: even a wrongly triggered cleanup just misfiles something, and it’s a two-line change. It doesn’t stop the underlying bug, though. A session’s manifest can still vanish from where every tool expects to find it, active, mid-task. It’s a good second layer and no substitute for the first.

What ships

A ticket is filed for the deletion itself. We’re extending the desk lease’s stale-window pattern to manifest writes. Before any close routine deletes or overwrites another session’s manifest, it takes a per-file lease keyed on last_touch, with the same 30-minute window and the same atomic-write discipline that already protects the desk. Given how many sessions are running against this repo at once on a given day, that path needed the same care as the one that stops two agents from fighting over a browser window.

If several automated sessions share one store, check whether any cleanup routine can delete another session’s records without confirming they are idle. Version control can restore a deleted file, but only after someone notices it is gone.