Two patches went out on the same day and came back in a single revert a few hours later. Both were aimed at one staff admin page, the edit control that moves a customer account onto a different configuration. That change touches a schedule format, a set of category mappings, and every downstream report that reads them back out. The control had always run through an older, narrower code path that had never been pointed at the platform’s migration engine, the same engine used elsewhere for bulk moves. Wiring it up looked simple.

The first ship, and the one that broke it

The first patch connected the control to the real migration engine. It had a preview step that shows exactly what will change with nothing written, and a separate confirm step that commits it. It shipped clean.

Thirty-five minutes later, a second patch added the piece the preview needed in order to render: a shared dialog library, pulled in for the first time on that admin page. Loading it also loaded its base stylesheet, and that stylesheet’s global resets collided with the page’s existing layout. The collision only showed on richer accounts, the ones with a full sidebar and a structure table rendering next to the edit controls. Our pre-ship check had only run against a thin, mostly empty test account, so it never saw the problem. There was nothing on that account for the reset to visibly break.

Why the revert took both patches

We pulled both patches back together. Reverting only the styling change would have left the configuration control pointed at a preview dialog that needed CSS we had just removed. On a real account it would have silently done nothing: no error, no result, just a control that didn’t respond. Taking both back was the only undo that left the page working.

The rebuild, same afternoon

The rebuild kept the same shape: the real migration engine, a preview, then a confirm. This time the report rendered inside a dialog widget that every other edit control on the page already used. No new library, no new stylesheet, nothing added to the page that wasn’t already there. A missing or malformed confirm flag meant the action was always treated as a preview and never wrote anything.

Independent review ran before any of it shipped again, and it caught three more problems:

  • A filename problem. The freshly compiled bundle reused the exact filename an earlier, reverted attempt had used, with different bytes underneath. A browser holding the old cached copy would never know to fetch the new one.
  • A comparison bug. The check for whether an account needed the guarded migration path compared a stored value case-sensitively against a lowercase constant. An account whose value was stored in a different case would fall through to the old, untested path.
  • A messaging bug. The confirmation screen said the conversion was complete even when the account write had succeeded but a secondary step, moving a folder of photos, had failed and returned a warning. A customer could land on a converted account with a page saying everything worked while some of their content was missing.

The error that came from code that didn’t contain it

Verification then reproduced the exact error the whole rebuild existed to eliminate. It was thrown from code that plainly didn’t contain what it was complaining about.

We got the explanation wrong the first time. A 231-line source rewrite had landed as a two-line change in the compiled bundle. We read that as proof the bundle hadn’t recompiled, withdrew the ship recommendation, and had the build re-verified byte for byte before trusting it again. That reasoning didn’t hold. A minified bundle collapses to one line, so a full rewrite of it will show as a two-line diff whether or not the content changed. We had treated a diff artifact as evidence.

The real cause had nothing to do with the compile. An earlier same-day attempt had compiled to that same output filename with completely different bytes inside, and the browser doing the verification still had that file cached. Because the filename hadn’t changed, it never asked for a fresh copy. It kept running the old, broken version against the new server code. Pulling the served file directly and checking it byte for byte showed the code had been right all along. Bumping to a filename nothing had ever used, and confirming the match again, ended it. The defect was a collision in what the fix was named.

Four days later, a lock that wasn’t held

The tool sat, reviewed and ready, until the first real run against an actual account. That was the first attempt to write anything for real, not a preview. It aborted immediately, claiming another conversion was already running against that account. Nothing else was running.

Two separate bugs had stacked up. The locking call required conditions that were never true in this code path, so it returned an error every single time, whether or not anything else was running. Separately, the code reading that result treated every non-success code as contention, when only one specific code means that. A real bug had been diagnosed as a busy lock because nobody checked which failure code had actually come back.

We fixed the lock call and the way its result was read. The same real account then ran cleanly, with nothing written until confirm and nothing left ambiguous once it was.

The tool merged into the main branch a few days after the ship-and-revert day. Getting there took a rebuild, three bugs found in review, one cache collision mistaken for a regression, and one locking bug mistaken for a concurrent run.

What I’d check next time

Test a change against a full, realistic account as well as an empty one, because the styling collision only appeared on accounts with a full sidebar and a structure table. When a fix is rebuilt, give the compiled file a name nothing has used, so browsers don’t keep serving an old cached copy. And check which failure code actually came back before calling an error contention.