The Green Script That Passed Against V3 While Production Ran V4

The hub showed six sub-sessions still running. Five of them had closed their tickets over an hour earlier. Nothing was hung and nothing had crashed. The dashboard just hadn’t noticed.

This was day one of /hub, a skill built to turn one chat window into a doorway for the whole session fleet. You spawn a sub-session, see which ones are working or waiting on you, and answer or cancel any of them without hunting through a dozen open tabs. The build passed review cleanly. It still broke in production four separate times before the day was out. The ticket tracking that day’s build ran six hours forty-four minutes, through four hardening passes forced by live failures.

The first failure set the pattern for the rest.

A marker that outlived its clone

Each dispatched clone writes a CURRENT_TASK marker when it picks up a ticket. The hub read that marker to decide whether a clone was busy. On paper that is reasonable. The marker is cheap to read, and the process doing the work writes it.

The catch is that the marker only gets cleared as the very last step of closing a ticket. If anything hiccups during close, even a network blip on the final write, the marker survives the session that wrote it. The clone is gone and the marker isn’t. The hub reads a ghost and reports the seat as occupied.

We traced it by pulling the clone list directly and diffing it against what the hub was showing. Five entries had markers pointing at tickets that were already closed and verified in the issue tracker. The clones had exited cleanly. The hub had no way to tell “still working” from “finished but nobody swept the marker.”

The fix was to stop treating the marker as ground truth and check the clone’s actual state, plus a reconcile mode that clears markers confirmed stale against that live check. It took twelve minutes of build time, was verified against the real clones, and the ticket closed. It was a small fix, but it was the same bug we kept re-finding in different clothes for the rest of the day: something that looked authoritative on disk had drifted from what was actually true.

The wrong turn: verifying a UI like an API

Earlier that week, for a separate integration project, we had built a script that hit the live endpoints directly. An unauthenticated read was refused (401), a forged token was refused (401), and the OpenAPI spec was served and correctly shaped. The output was clean pass/fail, diffable, with no ambiguity. It worked well enough that we reused the instinct on the next ticket that shipped: read the diff, run a script that checks response shapes, call it verified end to end.

That is right for an API. It is wrong for anything with a UI. The build was correct against its own plan, the diff matched the ticket, and the script exercising the relevant endpoints came back green. We closed the pass.

The problem was one layer up. The flow the code was built against was V3, and a separate merge earlier that day had already moved the live app to V4. Nothing in the diff flagged it, because the diff was internally consistent. It pointed at a page that no longer existed in the shape the code assumed. No test caught it either, because the tests were written against the same V3 assumptions as the code. A script that gets a 200 from an endpoint doesn’t know that the button a real user clicks now routes somewhere else.

We caught it only because we opened the live page, clicked the button, and watched it not do what the ticket said it would. We rebuilt it onto the real V4 flow. That time we didn’t close it on a diff read. We closed it after driving a browser against the live URL and confirming what a person would actually see.

Two rules out of that day

We now apply two rules to any session doing UI work, not just the hub.

Ticket-is-hypothesis. A shipped fix is a hypothesis about what’s true in production until something checks the live artifact and confirms it. A green diff and a passing script are evidence for the hypothesis. They are not the confirmation.

Literal item-close. A ticket doesn’t close on “the code that should produce the right state was merged.” It closes on “the actual state was checked, live, in the form a user or the next system would encounter it.” For an API, that means a request against the real endpoint. For anything with a screen, it means a browser hitting the real page, not a mock of it and not last week’s version of it.

By the end of that day the hub itself did the second half of that check. Instead of trusting that a merged fix implied a working flow, it drove a live browser pass against the real page when closing certain classes of ticket, the way that ticket got closed the second time. Five other tickets shipped that day the same way, verified against the running system rather than against the code that was supposed to produce it.

All four hardening passes came down to the same bug. Something durable and cheap to check, whether a marker file, a script response or a diff, was standing in for the thing that was true right now. Cheap checks are fine as a first signal. They stop being verification the moment they replace looking at the real thing.

dxdev LESSON · paste into your AI coding agent
LESSON: Verify UI Work in the Live Browser, Not Only Against Diffs and APIs
SOURCE: dxdev.com/blog/ui-verify-driven-browser-not-diff
WHAT HAPPENED: The hub reported five finished clone sessions as still busy because it treated leftover CURRENT_TASK markers as authoritative even after the clones had exited. Comparing the live clone list with the dashboard exposed the stale markers, so the fix added actual-state checks and reconciliation. Separately, a UI ticket passed a diff review and endpoint script, but both reflected an older V3 flow while production had already changed to V4. Opening the live page and clicking the real button revealed that the user flow did not work as the ticket required. The work was rebuilt for V4 and closed only after a browser pass against the real URL.
THE RULE: Close UI work only after exercising the live user flow in a browser against the current production page. Treat marker files, code diffs, mocks, and endpoint responses as supporting signals, never as substitutes for checking the state users actually encounter.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. For every UI ticket, does the close record include a browser check against the live URL and the intended user action?
      2. Does the verification target the currently deployed flow rather than a version or mock assumed by the code and tests?
      3. If a durable local marker or script result disagrees with live system state, does a reconciliation or live-state check determine the outcome?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.