A Staging Test Passed Two of Three Checks, and Both Passes Were False

Four copies of one customer email landed in an inbox. The trace led back to our own tooling, and the fix took a staging round trip. Along the way a staging smoke test reported passes that were false.

What caused the duplicates

Our weekly site checker signs in as staff and checks links on the live site. Its filter for links to skip was a denylist with no report or monitor terms in it, so it followed “run this report now” links, and each of those requests actually ran the report. The analysis records 161 duplicate customer emails across two days out of 996 examined. We inverted the filter to an allowlist, added tests that failed before the change and passed after it, and disabled the scheduled task.

That fixed the trigger, not the weakness. The report starter in dbo.monitor did a read (“is a run already open?”) followed by a separate write, with no lock, so two runs of the same report type could both see nothing open and both start. An atomic claim was added around it.

The fix that broke the page

We moved the lock to transaction scope, opened with ADO’s BeginTrans. On staging the staff “RUN with raw data” page broke every time. The error said the lock could not be acquired because of a timeout, which steered readers toward a contention theory. Nothing was holding the lock. A diagnostic deployed to staging captured the real return code, -999, a parameter validation error, with the transaction count at 0 when the lock call ran. The lock needs an active transaction, and BeginTrans over our ODBC bridge had not raised the count before the batch ran. The message had labeled every negative code a timeout, and a diagnostic that names the wrong cause is worse than none. The corrected version uses a literal BEGIN TRANSACTION inside the same SQL batch as the lock call.

The smoke test that passed

The staging smoke test reported two of three checks as PASS. Those passes were false. Staging and production share the live database, and the dbo.monitor rows appearing every five minutes were written by production, still running the old code. Reading them as evidence that the new code worked is reading production’s health as proof of our change. Only the one test that failed had actually exercised the new code.

What we did not verify

The incident analysis says the corrected fix passed a staging retest, with fiveMin and earlyMorning runs at 12:15:39, 12:15:42 and 12:15:45 all finishing, and that it had not yet shipped to production. By the rule above, those retest rows also need a named writer, and the record does not name one. The session close note says the fix is live on production and confirmed working. A filtered unique index, meant as a second layer, was written but not applied.

Check this tomorrow

Take your last “it passed on staging” result. For each row or record counted as evidence, find which process wrote it. If staging shares a database with production, any check that only looks for a row is measuring production.

dxdev LESSON · paste into your AI coding agent
LESSON: A passing row on a shared database proves nothing until you know who wrote it
SOURCE: dxdev.com/blog/race-condition-site-crawler-automation-side-effect
WHAT HAPPENED: A fix for duplicate report runs was tested on staging. The smoke test reported two of three checks as PASS, but staging and production share the live database, and the rows those checks counted as evidence were written every five minutes by production running the old code. Only the one failing check had exercised the new code, and it exposed a real break: the lock call returned a parameter error because the transaction the code thought it had opened did not exist yet.
THE RULE: A row existing is not evidence that a specific code path produced it. Name the process that wrote the row before calling it proof.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any smoke test or staging check that passes because a row or record exists, without identifying which process or environment wrote it.
  2. Any staging environment that shares a database, queue or mail configuration with production.
  3. Any error message that names a cause (such as a timeout) for every negative return code instead of reporting the real code.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.