The Batch-Archive Change That Returned 200 While Leaving Orphaned Data

The console popped up first. The crawl script was shelling out to git with no window-hiding guard, and every so often a terminal would flash on screen mid-crawl. That bug got fixed at 9:07 PM. The crawler itself came two hours later.

The idea was simple. Before anything ships to production, walk every admin page with a staff-authenticated session and see what breaks. That means an actual headless browser hitting actual staging URLs in sequence, the same way a real login would. It is not a unit test and it uses no fixtures.

The console leak

We already had a headless-guard test in the suite for exactly the console-leak class of bug. Once the git-shell issue was caught, verifying it was fast: run the existing guard against the fixed script, confirm no spawned window, commit, push.

The crawl that came back clean

The crawler took about two and a half hours to get right. The first pass authenticated once, then round-tripped through the sitemap and hit every admin route with a plain GET, logging status codes and any 500s.

It caught nothing interesting. Everything came back 200. A 200 on a page with a broken form action tells you nothing about whether the form works.

I had built this instead of leaning harder on the existing test suite for a reason. The 6,000-test suite we run pre-close was already a known pain point that day, and a separate ticket existed because pre-close was running the full suite instead of a targeted subset. But the status-code crawl was the version of “automated” that gives you a green check for nothing. We ran it anyway, got a clean report, and almost called it done there.

Reading the response bodies

It wasn’t done. A clean crawl on GET requests doesn’t tell you whether a batch operation is wired to the wrong table. So the second pass went deeper. Instead of only requesting each page, it inspected the response body for known failure signatures (missing table data, broken redirects, stale references) rather than just the HTTP status.

That was the regression class we cared about. An AI code-review pass had flagged it the same evening on a related admin batch-archive change. New tables were being copied and remapped, but the delete-original step wasn’t wired to match. That is a correctness bug that would return 200 all day while leaving orphaned data behind. Nothing throws and nothing 500s. It just does the wrong thing under a green light.

What the rebuilt crawl found

The rebuilt crawl found one real regression, filed as a single ticket, in staging, before it touched production. Everything else it flagged was pre-existing broken admin pages. They had been broken for a while and nobody had noticed, because nobody was walking that part of the site regularly. Those got filed separately so they didn’t drown out the one that mattered.

The whole run, first crawl to second crawl to filed ticket, took two hours and thirty-eight minutes, from 8:59 PM to 11:37 PM. After that it got scheduled: a weekly Tuesday check, same authenticated crawl, same signature-based inspection instead of status-code-only, running against staging before anything moves to prod.

The browser automation is plumbing. The difference was the shift from “did the page load” to “does the page do the right thing.” That meant reading response bodies for the specific failure signatures a real regression leaves behind instead of trusting a 200. The first version of the script would have run every week and reported all-clear on a bug that was about to ship. The second version is the one still running.

If you build a check like this, make it confirm that a page does the right thing and not only that it loads. Run it on a schedule against staging, and file pre-existing breakage separately so it does not bury the new problem.

dxdev LESSON · paste into your AI coding agent
LESSON: Verify Response Behavior Signatures, Not Just HTTP Status, Before Deploy
SOURCE: dxdev.com/blog/automated-regression-detection-crawl
WHAT HAPPENED: A pre-deploy crawler authenticated as staff and requested every admin route, logging status codes and 500s. Every page returned 200, and the crawl was nearly accepted as a passing pre-deploy check. A same-evening code review had separately flagged a batch-archive change where new tables were copied and remapped but the matching delete-original step was not wired up, a bug that would return 200 on every request while silently leaving orphaned data behind. The crawler was rebuilt to inspect each response body for known failure signatures (missing table data, broken redirects, stale references) instead of only the status code, and the rebuilt version caught the exact regression in staging before it reached production.
THE RULE: An HTTP 200 only proves a request completed, not that the underlying operation did the right thing. Before trusting an automated pre-deploy check, confirm it inspects response content for the specific failure signatures a real regression leaves behind, not just whether the server returned success.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Does any pre-deploy or regression check treat a 200/success status code as sufficient evidence that an operation behaved correctly?
      2. For write or batch operations, is there a check that inspects the actual data or response body for signs the operation was only partially applied?
      3. Are automated checks re-run against a rebuilt or expanded signature set after a related bug is found, rather than trusting the first clean pass?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.