--- title: "Staging Had Twelve Players Per Group and Production Had 47" canonical: https://dxdev.com/blog/release-after-release/ datePublished: 2026-03-31 --- Three production version tags in two days, two of them hotfixes. The initial release shipped the previous evening. By the following afternoon two more patches had followed, both merged and propagated across the full fleet of repos, both caused by real customer data finding conditions that staging never covered. March 31 registered zero sessions in our formal work log. The commit graph registered dozens of commits across the full repo fleet. That gap is worth understanding. The initial release closed a feature cycle: new admin behavior across pages that serve live operators. The merge went clean. The CI server deployed without complaint. We tagged it and felt the specific satisfaction of a thing done. On an older web application with live data behind it, that feeling and the actual state of the system are two different things. ## The cheap bug and the expensive one The first ticket that day covered event-folder image heights rendering inconsistently across admin screens. Wrong CSS class, wrong expected height. We fixed the class, verified in the browser, and merged. That is the cheap kind of bug: visual, self-contained, verifiable in two minutes on any environment with the right page open. A representative staging page can catch that class of problem before release. The second ticket was the expensive kind. Event roster and coaches pages were breaking after the initial release, but only under live data. Ordinary data, the kind any event director generates when importing a roster from a spreadsheet or assigning coaches across multiple groups in a season. The roster page had one failure mode. The coaches page had a different one. They showed up together on March 31 because production is where live data lives. In staging, our test events are clean: a handful of groups, one coach each, twelve players per group. The right schema and the right structure, but not the right variety. A production event might have three co-coaches sharing a single group, a roster import with 47 players where the previous import had 23, or an event folder linked across four competitions over six months. Any of those shapes can hit a code path staging never exercised. Both of those shapes hit the second ticket that day. Staging exercises known flows and representative fixtures. It cannot reproduce the full variety of historical data and operator behavior. Production monitoring exposed the shapes our fixtures had missed. ## The roster first, then the coaches We found the roster issue first, by walking the staging event pages with data reconstructed from what we could read in the production error logs. The error message gave us the group count that triggered the failure. The fix was targeted: one legacy page, one rendering path, one conditional that had not accounted for a count above a threshold no staging test had reached. We tagged the first patch. Done, we thought. That lasted about an hour. Then we opened the coaches page. The coaches failure was a separate defect. Same general surface area, different code path, different data condition. We had not known it existed when we shipped the first patch. Had we known, we would have held the first fix and merged both together. We did not know, so we did not hold, and the second patch followed within hours. We missed the coaches bug because we were working in the order the errors surfaced. The roster page was louder in the log, the coaches page less prominent, and we had not yet looked at coaches. When errors arrive asynchronously from real user behavior instead of from a test suite running in order, you find what you look for first. ## The loop that fills a release day The work session log shows zero entries for March 31. That is technically accurate. We were not running formal Claude Code sessions tracked by the session logger. We were in Jira, in the browser, in the error logs, in the code editor, in the terminal, and that is not how our session-tracking system logs time. The broader activity record showed we used AI tools throughout the day, just not through the instrumented path that writes to the work log. The agents helped in the narrowing phase of each cycle. Given an error trace and the relevant legacy page, they identified a likely defective condition faster than a cold read through a long legacy file. That compressed one leg of the diagnostic cycle. They did not eliminate the loop. The loop was: read the error log, form a hypothesis about which page and which condition, ask the agent to help narrow it, verify the hypothesis against staging with reconstructed data, write the fix, deploy it, read the error log again. That cycle ran twice on March 31, once per hotfix. A production error log is not a stack trace from a test harness. It shows the failure, the affected URL, and the session context. It does not show the data state that triggered it. You reconstruct that from the URL parameters, the session identifiers, and whatever the message captured about the offending value. For the roster bug, the count was in the message. For the coaches bug, the assignment structure was visible in the trace. Both gave enough to narrow the fix. Neither gave enough to stop the next bug from hiding until we looked for it explicitly. An agent can accelerate that investigation, but it cannot replace observing the relevant behavior and data unless it has access to those systems, and the work still includes verification. ## Dozens of commits for two bug fixes The main codebase runs across a fleet of [long-lived git clones](/blog/branch-graveyard-automated-cleanup/), and every hotfix propagates to all of them. Operators are isolated at the IIS level, each running against the same codebase in its own instance. A fix has to reach every relevant instance consistently, or the support surface varies by operator, which is a different operational problem. A propagation script handles the fan-out. The cost is the commit count. Two hotfixes, a handful of commits each across the merge, tag, and surrounding bookkeeping, fanned out across the full repo fleet. The aggregate looks alarming. It is what normal looks like in this deployment model. It also makes time accounting strange. Dozens of commits register as a substantial day in the git history, but the actual debugging and fixing took closer to four hours across two separate problem spaces. The rest was propagation overhead, the cost of a deployment architecture designed before anyone thought in terms of Git workflows. ## The side thread There was a second strand of work on March 31: a background investigation into a [Cloudflare migration](/blog/thousands-of-domains-to-a-layer-above/) and a bot-attack review. It is in the record, and it would be dishonest to omit. It stayed in the background. We were watching a live site through a stressful post-release day and checking the attack surface while waiting for things to stabilize. The Cloudflare thinking lived in a tab while the hotfix loop ran. The migration is a longer project with its own arc. On a day with two hotfixes in flight we were also watching the perimeter, and the perimeter stayed quiet. ## What the tag means now We used to treat the tag as the end: ship it and the work is done. That frame kept producing the same experience. A release felt finished, then it was not finished, then we felt behind. Now the tag is the moment real data gets its first vote. Staging covers the paths you anticipated and production covers the ones you did not. A mature platform accumulates data variety no staging environment can replicate. Event directors import rosters from Excel files with inconsistent column ordering. Co-coaches appear in configurations we would not have thought to test. Event folders accumulate relationships that span multiple seasons. We only learn which edge cases survive the first release by watching what breaks in the first twenty-four hours. So the day after a non-trivial release is not a rest day. We stay close to the error logs on day one, scope each fix tightly, and ship the next patch as soon as we know we need one. Not at the end of the week, and not after a clean window appears. The second patch shipped the same afternoon as the first, and that is the part we feel good about: it did not sit while we recovered from the first. A release is complete when the agreed checks have passed against live-shaped data and the system is stable enough for its stated risk level. Define the watch window, name the high-risk flows, and decide who reads the error logs on day one.