A Titleless Event Entry Vanished From Production 71 Minutes After Deploy

A titleless entry showed up in an event listing, and 71 minutes later it was gone from production. The read-side guard did that, not the delete fix. It dropped the broken row from the listing the moment it deployed, and no data changed.

What the ghost was

The symptom was an entry card with no title, no accounts and nothing to click into. It was a rectangle on the page, and it looked like a rendering bug.

It wasn’t. Event sites and entries are joined through a linkage row. The entry’s display fields come from the site, so the row only knows “site X owns this entry.” When someone deleted an event site, the delete removed the site and its own children, but it never touched that linkage row. The row stayed behind pointing at a site that no longer existed. The listing query walked the links, found a row, and rendered an entry whose title lookup returned nothing.

The diagnostic path was short. An empty title field pointed at a missing join target. A missing target meant something had deleted the parent without cleaning up the children. The delete path for events was the obvious suspect, so we read it, and the linkage table was not in the list of things it cleaned up. One missed backref was the whole bug.

Guard first, delete path second

There were three fixes on the table:

1. Stop the render. The listing should skip any link whose site no longer resolves.
2. Stop the leak. The delete path should remove the linkage row along with the site.
3. Clean up. A stored-proc-level prevention plus a one-time sweep for rows already orphaned.

We shipped 1 and 2 in the same deploy. The order matters, because only the first one touches rows that already exist. Fixing the delete path alone stops new orphans, but every row created before the deploy keeps rendering a ghost until someone cleans the data. The read-side guard has no such dependency. It treats “link points at nothing” as “nothing to show,” so it hides every existing orphan the moment it deploys.

The shape of the guard is small. Instead of trusting the link and hoping the site is there, the listing requires the site to resolve. In SQL terms that is an inner join from the linkage row to the site, where the old path effectively left-joined and rendered whatever came back. A row whose site is gone drops out of the result set before the UI ever sees it.

The part we did not finish

Item 3 did not get done. The stored-proc prevention, which enforces the cleanup at the database layer so no future delete path can forget it, and the one-time cleanup of existing orphans both need the production database. We were on an unstable connection, and we were not going to run a destructive sweep against production over a link that might drop mid-statement. If it dropped, we would be left with a half-applied cleanup, which is worse than an unapplied one.

So we parked it. Both changes are staged, written up in the ticket, and set to resume from a stable network. That is the cost of this session: the fix is complete on the read side and incomplete at the source. The orphaned rows are still sitting in the table. They are invisible now, but they exist.

Why we would ship it in this order again

A source fix answers “how did this happen.” A read-side guard answers “what does a user see while the data is still wrong.” Those are different questions, and production is always in the second situation before it gets to the first. Data that was orphaned last month, by a code path someone has since rewritten, is exactly as broken as data orphaned today. If the only defense is that nothing bad ever gets written, the first bad row that slips in becomes a visible defect.

The guard also holds up against the next bug of the same family. The delete path we fixed was one of possibly several ways to remove a site. We found the one that produced this ghost, and we cannot claim we found all of them. A listing that refuses to render dangling links is protected against the ones we did not find.

What it does not do is fix anything. The guard is a mask over bad data, and if it is the only thing that ships, the table fills with rows nobody can see, and the next person to write a query against it inherits a mess. That is why the parked work matters, and why the ticket stays open until the sweep runs. The guard buys time, and the cleanup pays it back.

Check the screens when a record can be deleted from more than one place

When a record can be deleted from more than one place, check whether the screens that display it still work if a dependent row is left behind. Treat the guard as a stopgap: the orphaned rows still exist, so keep the cleanup ticket open until the sweep has run.

dxdev LESSON · paste into your AI coding agent
LESSON: Guard Read Paths Against Dangling References Before Repairing Their Source
SOURCE: dxdev.com/blog/orphaned-rows-phantom-ui-defensive-guards
WHAT HAPPENED: Deleting an event site removed its own children but left behind the row linking that site to an entry. The listing followed the dangling link and rendered an entry whose title and accounts could no longer be resolved. The deployed listing change requires the linked site to resolve, so existing orphan rows immediately disappear from the UI. The delete path was also updated to remove the linkage row, preventing new orphans through that path. A database-level safeguard and cleanup sweep were deferred because production access was unstable, leaving the hidden orphan rows in place.
THE RULE: When invalid historical data can reach a user-facing read path, make the read path reject unresolved references immediately while separately fixing and cleaning up the write-side cause. Keep cleanup work open: a visibility guard contains bad data but does not repair it.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Does every user-facing query that follows a linkage row require its referenced parent record to resolve before rendering the result?
      2. Does every supported parent-deletion path remove or otherwise handle dependent linkage rows?
      3. Is there a tracked, safely executable plan to identify and remove existing orphaned linkage rows after the read-side guard ships?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.