---
title: "The Hotfix That Shipped at 3:36 PM Without a Repro"
canonical: https://dxdev.com/blog/targeted-hardening-without-repro/
datePublished: 2026-06-25
---
At 3:36 PM, we shipped a release because one organization-account event was stuck and the exact trigger would not come back on demand.

Reproduction is the gold standard for a bug report. It tells you what happened, lets you prove the fix, and gives you a test that guards the next regression. We want it whenever we can get it. But a failure you can't reproduce can still be fixable. It can mean the failure depended on state we no longer had, an intermittent sequence, or a route through code that should not be available in the first place. Here, the useful work was removing the reachable condition that could produce the kind of state we had just seen.

## A stuck event is evidence

The reported failure was concrete. An event on an organization account was stuck, and we could not recreate the action sequence that put it there.

That left two separate questions, and treating them as one would have slowed us down.

The first was forensic: what precise sequence got this event stuck? We had no reliable answer. The report never turned into a repeatable local case, and we were not going to invent one in the ticket to make the story tidier.

The second was architectural: what state or mode is still reachable in the organization-account event path that should no longer be? That one had an answer. The deprecated preview mode could still be forced onto an organization-account event.

We do not know whether that was the historical trigger for the reported event. The record doesn't establish it. What it establishes is that the path was still open, it was deprecated, and closing it protected that account class either way.

That is enough to act when the change is narrow and the failure mode is bad. It is also a bet. We signed off on a hotfix without the thing we usually want most, so the actual cause may still be out there, waiting for the next stuck organization-account event to look identical to this one and prove us wrong. We would rather ship a constraint we can explain than sit on a comfortable silence while the deprecated path stays open.

## Making the invalid route unavailable

The production change hardened the organization-account event path so that events can no longer be forced into deprecated preview mode.

It is a constraint at the boundary where the old mode could still be selected or applied, with no broad rewrite of event setup. An organization-account event must not enter that mode, whatever input or stale state tries to push it there.

That beats a patch that only repairs the reported event. A one-record repair might relieve the incident, but it leaves the system able to make the same bad choice again. Blocking the bad state in the code path turns a conditional warning into an invariant.

> An invariant means this class of object cannot enter this state through this path.

We kept it targeted on purpose. One stuck event, an unreproducible trigger, and a deprecated mode that should not have been reachable for this account class did not justify rewriting every event transition or touching unrelated account paths. It justified closing the door that should already have been shut.

## The reproduction hunt we skipped

We could have kept the incident open until we had the exact clicks, timing and data state. That is the cleanest debugging narrative, and it was the wrong release criterion here. A perfect reproduction would have told us more, and might have given us a highly specific regression test. But waiting for it would have kept a known-deprecated mode reachable while we chased an intermittent condition we could not observe again.

The other bad option was to treat the event as an isolated data problem. That would make the current incident disappear without changing the condition that made it possible. A report about one event does not mean the defect belongs to one event.

So we did not fix without evidence. The evidence was the stuck event plus an accessible deprecated mode in the affected code path. We fixed what we could prove and stayed precise about the part we could not.

## Keeping the other report out of the hotfix

The hotfix did not become a bucket for every event problem observed that day. A separate intermittent blank single-competition setup page went into its own item.

A vague report tempts you to join failures because they happen near the same feature, and that turns diagnosis into narrative. The stuck event was addressed by preventing deprecated preview mode on organization accounts. The blank setup page stays its own intermittent problem until it has its own mechanism and fix.

The hotfix took about 2 hours and 30 minutes from investigation through shipment. At the end we still had no faithful reproduction of the original report. What we had was a narrower system: an organization-account event could no longer be forced into a mode we had already deprecated. When the old path should not exist, removing it is the fix.