Two customer logins broke the same week, with the same symptom both times: the user is logged in but looking at the wrong group. The root cause was the same too, a stale role row left over from last season in the table the login path walks. If you stop reading at “same root cause,” you take the first fix you wrote, run it against the second account, and destroy the wrong record. We almost did. We stopped because we made ourselves look at the actual rows for each account before reusing anything. The two shapes differed, and the fixes were exact opposites.
The shared root cause
The product is a SaaS platform. People belong to groups through role rows: this person is a member of that group, this person has a lead role on another one. At login the app walks the person’s role rows to decide which groups to show and which one to land them on. That design attracts stale data, because three real-world events all mutate the same table. A member moves to another group. A member legitimately belongs to two groups in the same organization. A new season rolls membership forward. Any of them can leave a role row pointing somewhere it shouldn’t.
So when two accounts showed up with “wrong group on login,” the cause matched: a role row from last season that the login walk still honored. A lot of triage ends there. You’ve found the cause, you’ve already written the repair, you move on.
“A stale role row” names a category. The change you need depends on what the rest of that account’s rows look like, and those differed.
Shape A: moved between groups, needs the row re-pointed
The first account was a member who moved between groups. Their login routed them to their old, now-inactive group. When we pulled their role rows, there was exactly one relevant row, and it still pointed at the old group. The new group’s member record existed. The role just hadn’t followed them over.
The fix is a re-point. You take the existing role row and update it to reference the new group’s member record. You delete nothing, because there is no extra row to remove. The row is right in kind and wrong in target. Delete it and they lose their membership entirely.
The re-point was more than a one-column update. Their email-confirmed status lived on the old member record, and the new record showed unconfirmed, so the fix also had to carry the email-confirm timestamp forward. That is where a datetime gotcha lived: reading the date into the application layer and writing it back stringifies it into a format the database refuses to parse on the way in. The fix was to never let the value leave SQL, copying it column to column with a correlated subquery. That gotcha is its own post. Here, what matters is that Shape A’s whole repair is “update what exists, carry one field forward, delete nothing.”
Shape B: two live rows, needs the stale one deleted
The second account had the same symptom and the same root-cause category: the login showed the wrong group. The row picture was different. This person had two role rows, a lead role on a group from last season and a member role on a current group. Both surfaced in their account, and the stale lead role was the one polluting the view.
The fix is a delete. There is nothing to re-point, because the current row is already correct. The problem is the presence of the extra, stale row. You remove the lead role, leave the orphaned record behind for an admin to clean up, and you’re done.
Lined up:
- Shape A: one row, wrong target. Fix = re-point it. Delete count: zero.
- Shape B: two rows, one current and one stale. Fix = delete the stale one. Update count: zero.
The repairs are opposites. Shape A’s repair never deletes, and Shape B’s never updates. Had we run the script we wrote for the member who moved against the two-row account, the re-point logic would have aimed at the wrong row and rewritten data that was already correct, while the actual stale row stayed right where it was. Same symptom, same root-cause label, and a repair that would have made things worse.
Why the symptom and the root cause both lie to you
The parts that matched were the symptom and the root cause. “Wrong group on login” matched. “Stale role row” matched. Every field you’d put in the title of the bug matched. The part that differed only shows up when you query the rows for that account and look at their structure: how many there are, which is current, what each points at.
The pattern reaches well past role rows. Two null-pointer crashes with the same stack trace can have different upstream causes. Two “duplicate key” errors can need a merge in one case and a delete in the other. A category is not a repair plan. The plan depends on the surrounding data shape, and you only learn the shape by looking at the actual rows, which means the ticket and the last fix you wrote can’t tell you.
It is especially easy to get this wrong when you wrote the first fix yourself, ten minutes ago, and it’s right there. Reuse is the path of least resistance, and it feels responsible. Reusing your own correct, tested code is nothing like copy-pasting a stranger’s Stack Overflow answer. It was still the wrong move here, because the code was correct for a different shape.
What we did instead
Each account got its own one-shot repair script, and each script asserted its own shape before touching anything. We did not run the same script with a different ID, since that move is what conflates the two. Separate scripts.
Both followed the same defensive skeleton, which is worth stealing on its own:
- Dry run by default. The script reports what it would do and writes nothing unless you explicitly pass a commit flag. The default invocation is safe.
- Assert the shape before any write. Confirm the person exists. Confirm the records you expect actually do. Confirm the last names match the name on the ticket so you know you have the right human. Then confirm the row layout. Shape A’s script asserts the role row points at the group it expects before re-pointing it. Shape B’s script asserts there are two rows and that the one it’s about to delete is the stale one. On any mismatch the script writes its report and aborts. It refuses to run against a shape it wasn’t written for.
- Idempotent. If the repair already partially ran, the script detects the applied state and resumes from where it left off, with no error and no double-applying.
- Emit a before/after report. The output is a machine-readable snapshot of every row it touched, and it goes into the ticket as the evidence of what changed.
The assertions are the part that matters. Shape B’s script, run against Shape A’s account, fails its own pre-flight. It expects two rows and finds one, so it aborts and writes a report saying so, instead of deleting the only row the member who moved has. The script refuses to be the wrong fix. That turns “we have to remember these are different” into “the tool won’t let us conflate them,” and only the second version survives a tired Thursday afternoon.
A matching symptom and a matching root-cause label do not mean a matching repair. Before reusing a fix on a second account, query that account’s actual rows and confirm they have the shape the fix was written for. Then check that your repair scripts default to a dry run, verify the data layout before any write, and abort when it doesn’t match.