Every staff write in our admin center returned success:1 to the browser and changed nothing in the database. Comment on an account, gone. Flag an account, ignored. Change a customer’s email, change their password, change their sport, delete a record. The UI said it worked every time. None of it touched a single row.
The cause was one line a developer flipped in October 2023 to iterate locally, and never flipped back. It rode along inside a feature branch for about two and a half years, got merged in April 2026, and shipped looking like progress.
This is the postmortem. The bug is almost embarrassingly small. The mechanics of how it survived two and a half years and slipped past a deliberate cleanup commit are the part worth your time. The timeline: the branch merged into develop on April 24, the flag was live everywhere the handler ran from that point, the April 28 release carried it to production, and the same-day hotfix went out at 22:03 that night.
The symptom: saves that succeed and do nothing
It came in as a support ticket: staff and customers reporting that comments weren’t saving on the account details page. You add a comment, hit save, the page acts like it saved. Reload, and the comment is gone.
There is no error. No red banner, no spinner-of-death, no 500. The save returns success. So the first instinct of everyone in the loop, support, the customer, and at first me, is that it’s a refresh problem, a caching problem, a “you must have not clicked save” problem. The actual failure mode never announces itself, because from the browser’s point of view nothing failed.
And it wasn’t just comments. Comments were the one customers happened to notice and report. The same code path was silently dry-running Delete Account, Flag Account, Change Sport, Change Email, Change Password, Birthdays, and dozens of other write cases. The handler dispatches around sixty cases, the large majority of them writes, and every write returned success and persisted nothing.
The dispatcher: one flag gating every write in the file
The admin center routes most of its account writes through a single classic-ASP handler: the account-details AJAX handler. Line 10 of that file was, at the time of the incident:
var EDIT_DATABASE = false;The handler routes around sixty case labels, the large majority of them writes, and each one passes the flag down to its db object before calling through to execute the SQL. With EDIT_DATABASE = false, every UPDATE and DELETE the handler would normally run becomes a dry run. The code walks the same path, builds the same command, and skips the part where it actually hits the database.
The flag exists for a legitimate reason. When you are developing a destructive write path locally, you do not want to be issuing real UPDATEs and DELETEs against whatever data your dev box is pointed at while you iterate on the surrounding logic. So you flip it false, you build the feature, you verify the non-destructive parts, and then you flip it back before you ship. That last step is the whole game.
Why nothing screamed
The reason this was invisible rather than merely sneaky is structural.
The database object’s execute method does not throw on a dry run. With the flag off, the command simply doesn’t execute, and the function returns normally. The JavaScript on the page gets success:1 back over the wire. The SQL just never ran. No exception, no error log entry, no failed test, no nothing. The success response and the no-op are indistinguishable from the outside.
This is the core of the lesson, so I’ll state it plainly. A write path that cannot throw on a no-op has no immune system. If “did nothing” and “succeeded” produce the same response, then there is no signal anywhere in the stack, not in the UI, not in the logs, not in a test suite, that can tell you the writes stopped working. The only thing that can detect it is a human reading the database afterward and noticing the row didn’t change. That’s exactly the loop here. The flag was live the moment the branch merged on April 24, but it produced no alarm at any layer, so nothing flagged it until a customer reported a vanishing comment.
The archaeology: two switches flipped in one session
When I pulled the thread on where that line came from, it traced back to a feature branch started by another dev on October 13, 2023.
To iterate on it locally that day in October 2023, he flipped two debug switches in the same session. He commented out the try/catch around the page’s main request handler, so that errors would surface raw instead of being swallowed. And he set EDIT_DATABASE = false, so his local writes were dry runs. Both are completely reasonable local-dev moves. Both are the kind of scaffolding you put up at the start of a session fully intending to tear down before you ship.
Then the branch sat. For about two and a half years. It wasn’t abandoned, it was parked, the way real work gets parked when something more urgent walks in. The two debug switches sat parked with it.
The cleanup that found one switch and missed the other
This is the part that haunts me, because the dev did the right thing and it still wasn’t enough.
When that dev picked the branch back up and shipped it on April 24, 2026, his cleanup commit explicitly restored the try/catch. The commit body even called it out: the block “was commented out as debug scaffolding in Oct 2023, never restored.” He remembered there was scaffolding. He went looking for it. He found one of the two switches and put it back.
He missed the EDIT_DATABASE line. And here’s the cruel part: it wasn’t sitting next to the try/catch. The flag is on line 10 of the handler, at the very top of the file. The try/catch he restored is buried down around line 2160, two thousand lines away. Same file, same debt, planted in the same October 2023 session ten minutes apart, but at opposite ends of a 2,000-line classic-ASP file. He restored the one he had to scroll to. He never saw the one at the top, because two and a half years is long enough for “what’s temporary in here” to decay to zero, and memory is not a diff.
So the branch merged into develop with the dry-run flag still false. The April 28 release carried it to production, and the regression shipped.
The fix
The fix is the least interesting line in this entire story:
- var EDIT_DATABASE = false;+ var EDIT_DATABASE = true;One line, reverted in a same-day hotfix, shipped at 22:03 on April 28, the same day the release deploy carried the bug to production. The branch had been merged into develop since April 24, so the dry-run had been eating writes wherever the handler ran for days before that. The hard part was never the fix. The hard part was that a save returning success gave nobody a reason to look until a customer noticed their comment had vanished.
What I actually changed because of this
A one-line revert is not a takeaway. Here’s what came out of it.
Long-idle branches get a scaffolding scan before merge. The author who planted the debt is the worst-positioned person to spot it from memory, because their mental model of “what’s temporary in here” has aged out. So memory is out of the loop entirely. Any branch idle past a month gets diffed against its base, and the diff gets grepped for known scaffolding signatures before it merges: EDIT_DATABASE\s*=\s*false, commented-out try/catch blocks, Response.Write blocks added for inspection, hardcoded test usernames or emails, if(false) and if(true) gates, and any dry-run flag living outside a test context. Hits are discussion points, not auto-reverts. Some are legitimate. The point is that they get seen.
Remove scaffolding as a set, not line-by-line from memory. The deeper failure wasn’t forgetting one line. It was treating two switches planted in one session as two independent things to remember, months apart, instead of one set to tear down together. If you flip three debug switches at the start of a session, write them down as a group right then, in the commit or the ticket, so the cleanup is a checklist and not an act of recall.
On a legacy stack with no test harness, the diff scan plus a browser spot-check is your cheapest catch. A “save succeeds, row unchanged” failure can only be caught by a write-then-read assertion against a real database. We don’t have that harness on this code. What we do have is the diff, and a quick manual click through any write path the branch touched. That combination would have caught this in minutes, before the branch ever merged. It costs less than the cleanup did, by a wide margin.
Related
- Don’t Trust the Green Deploy: Grep the Live File for Your Ticket Marker: green pipeline, silent no-op, the habit that catches what the deploy dashboard won’t
- The “queue worker drains it” story was vapor: the create command said NOT YET IMPLEMENTED: same shape, a success return masking a no-op that nobody verified
- The Hotfix That Lied: STAGING/LIVE in JIRA, Never Merged to Master: a branch parked so long its status diverged silently from what actually shipped
- A 47,490-line JavaScript file and what it tells you about a profitable legacy app: the legacy codebase context that makes a flag at line 10 invisible from line 2160
- Before You Merge a Branch That’s Been Sitting for a Year, Grep It for Your Own Footguns: the process that would have caught this, diff + grep the stale branch before merge