---
title: "We Diffed Every Row and the Bug Was in the Script Order"
canonical: https://dxdev.com/blog/the-rollover-that-could-never-recover/
datePublished: 2026-08-04
---
The alert had gone quiet, and that was the problem. A rollover that hits a duplicate-key error is supposed to fail loud and then heal itself on retry. Ours failed loud once. Every retry after that failed the same way, and nobody noticed because a customer only calls in once.

## The step order

Season rollover runs as a sequence. It creates the new season's records, then runs a cleanup pass that clears duplicate-key collisions left over from a prior partial run. That ordering was backward. The cleanup that clears duplicates ran after the step that creates them.

So the first time a rollover hit a duplicate key, it threw. The retry hit the exact same duplicate, because nothing had run yet to remove it. The cleanup code existed and it was correct. It sat downstream of the failure it was meant to prevent.

Any site that hit this once could never rerun a rollover successfully again. Every future attempt was stuck until someone reordered the two steps.

## Looking in the wrong place

We went in expecting a data problem, something in the source rows for a specific site. A duplicate-key error is the database engine telling you a value already exists, so the natural move is to go look at the values. We pulled the failing site's season data and diffed it against a season that had rolled over cleanly. We found nothing structurally different. There were no orphaned rows and no schema drift, and nothing explained why one key collided and the other didn't.

We spent a while on that path, and it cost real time. The answer was in the rollover script's step order. The cleanup routine was defined, tested, and worked exactly as written, but it came after the create step instead of before it. A duplicate key thrown at create time meant cleanup never got a chance to run. Every retry hit create first, so every retry recreated the collision that cleanup needed to have already cleared.

## The fix and the two silent sites

I moved the cleanup pass ahead of the create step, so any leftover duplicate from a prior run gets cleared before the new season tries to write over it. It shipped to develop for 3.361.

Three customer sites were confirmed stuck by this, and only one had called in. The other two had been silently unable to complete their rollover, and nobody knew, including us. We ran the fix against production for the two that were straightforward, and both completed their rollovers end to end.

We didn't want to learn about the other two by phone again. So the last piece was a scan tool that walks sites looking for rollovers stuck in this specific failure state, so we find them before a customer does. It doesn't try to be clever about data, which is the lesson from the wrong turn. No amount of staring at rows was going to surface a step-ordering bug. The scan just checks whether a site is sitting in the stuck state this bug produces, because that's the only thing worth checking for.

A retry only heals a failure if the repair step runs before the step that fails. Here two of three affected sites were stuck with no one aware, and the only signal we ever got was one phone call.