Every staff write in our admin center returned success:1 to the browser and changed nothing in the database. Comment on an account, gone. Flag an account, ignored. Change a customer’s email, change their password, change their category, delete a record. The UI said it worked every time. None of it touched a single row.
The cause was one line a developer flipped in October 2023 to iterate locally and never flipped back. It rode along inside a feature branch for about two and a half years, got merged in April 2026, and shipped looking like progress.
The bug is almost embarrassingly small. How it survived that long, and how it slipped past a deliberate cleanup commit, is the part worth your time. The branch merged into develop on April 24, so the flag was live everywhere the handler ran from that point. The April 28 release carried it to production, and the same-day hotfix went out at 22:03 that night.
Saves that succeed and do nothing
It came in as a support ticket: staff and customers reporting that comments weren’t saving on the account details page. You add a comment, hit save, and the page acts like it saved. Reload, and the comment is gone.
There was no error. No red banner, no 500. The save returned success. So everyone in the loop, support, the customer and us, first assumed a refresh problem, a caching problem, or a “you must not have clicked save” problem. The failure never announced itself, because from the browser’s point of view nothing had failed.
Comments were only the case customers happened to notice. The same code path was silently dry-running Delete Account, Flag Account, Change Category, Change Email, Change Password, Birthdays, and dozens of other write cases. The handler dispatches around sixty cases, the large majority of them writes, and every write returned success and persisted nothing.
One flag gating every write in the file
The admin center routes most of its account writes through a single handler in the legacy application: the account-details AJAX handler. Line 10 of that file read:
var EDIT_DATABASE = false;Each of the roughly sixty case labels passes the flag down to its db object before calling through to execute the SQL. With EDIT_DATABASE = false, every UPDATE and DELETE the handler would normally run becomes a dry run. The code walks the same path and builds the same command, then skips the part where it hits the database.
The flag exists for a good reason. When you are developing a destructive write path locally, you don’t want real UPDATEs and DELETEs going against whatever data your dev box points at while you iterate on the surrounding logic. So you flip it false, build the feature, verify the non-destructive parts, and flip it back before you ship. That last step is the whole game.
Why nothing screamed
The database object’s execute method does not throw on a dry run. With the flag off, the command simply doesn’t execute and the function returns normally. The JavaScript on the page gets success:1 back over the wire. The SQL never ran. No exception, no error log entry, no failed test. The success response and the no-op look identical from the outside.
A write path that cannot throw on a no-op has no immune system. If “did nothing” and “succeeded” produce the same response, no signal exists anywhere in the stack, in the UI, the logs or a test suite, that can tell you the writes stopped working. Only a human reading the database afterward and noticing the row didn’t change can detect it. That is exactly the loop here. The flag was live the moment the branch merged on April 24, but it raised no alarm at any layer, so nothing flagged it until a customer reported a vanishing comment.
Tracing the line back
When we pulled the thread on where that line came from, it led to a feature branch another developer started on October 13, 2023.
To iterate locally that day, the developer flipped two debug switches in the same session. They commented out the try/catch around the page’s main request handler so errors would surface raw instead of being swallowed. And they set EDIT_DATABASE = false so their local writes were dry runs. Both are reasonable local-dev moves. Both are scaffolding you put up at the start of a session fully intending to tear down before you ship.
Then the branch sat for about two and a half years, parked the way real work gets parked when something more urgent walks in. The two switches sat parked with it.
The cleanup that found one switch and missed the other
This is the part that stings, because the developer did the right thing and it still wasn’t enough.
When they picked the branch back up and shipped it on April 24, 2026, their cleanup commit explicitly restored the try/catch. The commit body even called it out: the block “was commented out as debug scaffolding in Oct 2023, never restored.” They remembered there was scaffolding and went looking for it. They found one of the two switches and put it back.
They missed the EDIT_DATABASE line, and it wasn’t sitting next to the try/catch. The flag is on line 10, at the very top of the file. The try/catch that got restored is down around line 2160, two thousand lines away. Same file, same debt, planted in the same October 2023 session ten minutes apart, at opposite ends of a 2,000-line file. The one at the bottom got restored because you have to scroll to it. The one at the top was never seen. Two and a half years is long enough for “what’s temporary in here” to fade to nothing, and memory is not a diff.
So the branch merged into develop with the dry-run flag still false, and the April 28 release carried the regression to production.
The fix
The fix is the least interesting line in this story:
- var EDIT_DATABASE = false;+ var EDIT_DATABASE = true;One line, shipped in a same-day hotfix at 22:03 on April 28. The branch had been in develop since April 24, so the dry run had been eating writes wherever the handler ran for days before that. The fix was never the hard part. The hard part was that a save returning success gave nobody a reason to look until a customer noticed their comment had vanished.
What we changed because of this
Long-idle branches get a scaffolding scan before merge. The author who planted the debt is the worst-placed person to spot it from memory, because their sense of what’s temporary has aged out. So we took memory out of the loop. Any branch idle past a month gets diffed against its base, and the diff gets grepped for known scaffolding signatures before it merges: EDIT_DATABASE\s*=\s*false, commented-out try/catch blocks, Response.Write blocks added for inspection, hardcoded test usernames or emails, if(false) and if(true) gates, and any dry-run flag living outside a test context. We treat hits as discussion points and decide case by case, since some are legitimate. What matters is that someone sees them.
Remove scaffolding as a set. The deeper failure was treating two switches planted in one session as two separate things to remember months apart. If you flip three debug switches at the start of a session, write them down as a group right then, in the commit or the ticket, so the cleanup is a checklist you tick off.
On a legacy stack with no test harness, the diff scan plus a browser spot-check is the cheapest catch. A “save succeeds, row unchanged” failure can only be caught by a write-then-read assertion against a real database, and we don’t have that harness on this code. We do have the diff, and we can click through any write path the branch touched. That combination would have caught this in minutes, before the branch merged, and it costs far less than the cleanup did.
If a system reports success for a write, check that the data actually changed. A debug flag left on can make every save look fine while nothing is stored.
Related
- Don’t Trust the Green Deploy: Grep the Live File for Your Ticket Marker: green pipeline, silent no-op, the habit that catches what the deploy dashboard won’t
- The “queue worker drains it” story was vapor: the create command said NOT YET IMPLEMENTED: same shape, a success return masking a no-op that nobody verified
- The Hotfix That Lied: STAGING/LIVE in JIRA, Never Merged to Master: a branch parked so long its status diverged silently from what actually shipped
- A 47,490-line JavaScript file and what it tells you about a profitable legacy app: the legacy codebase context that makes a flag at line 10 invisible from line 2160
- Before You Merge a Branch That’s Been Sitting for a Year, Grep It for Your Own Footguns: the process that would have caught this, diff + grep the stale branch before merge