The Hotfix That Lied: STAGING/LIVE in JIRA, Never Merged to Master

The ticket said STAGING/LIVE. It had a fix version, 3.349, and a real commit with a real diff. And customers were still getting the broken behavior, because the branch that commit lived on had never been merged to master. Nothing in my flow caught the gap between “closed in the tracker” and “in production,” so the ticket sat there lying about itself until a fresh bug report dragged it back into the light.

This is a small failure with a big shape. If you run a tracker and a release branch and you trust the tracker to tell you what’s shipped, you have this hole too. Here is exactly how it opened on me, and the two rules I now gate on so it can’t open the same way again.

The report that didn’t match the tracker

The symptom came in as a new problem: emails were still going to Verizon’s vzwpix.com email-to-text gateway. That gateway is one of the old free SMS-by-email tricks, where you send to <number>@vzwpix.com and the carrier turns it into a text. Carriers are decommissioning these gateways, so the plan was to route the affected carriers through a Twilio opt-in path instead. There was a ticket for exactly that. It was closed. STAGING/LIVE. Done.

Ben’s question when the report landed cut straight to it: “these are serious issues, how was it marked as done but not pushed to master. thats what a hotfix is? i thoiught we confimed already.”

That is the right question, and it is the whole post. A ticket marked done is a claim. Until you check the claim against git and against the live site, it is exactly as trustworthy as any other thing someone typed into a form.

The dig: a commit that existed, on a branch that went nowhere

Here is what I found when I actually looked instead of trusting the status field.

The fix was for the carrier-routing ticket. The code change added three carrier IDs (Verizon, Xfinity/Straight Talk, and Spectrum Mobile) to the opt-in carrier map in the texting library, so those carriers would route through the opt-in path instead of the dying gateway. That change was real. It existed as a commit on the hotfix branch.

Then I looked at master. Master HEAD had no trace of the change. None. The hotfix branch was cut, the commit was made, and the one step that actually puts a hotfix into production, the merge to master, never ran. The branch was a dead end with a green ticket pointing at it.

So every layer agreed with itself and lied together. The tracker said LIVE. The branch said “here’s your fix.” And production said “still broken,” because production only ever sees what’s on master, and master had never heard of this commit.

The thing that makes this dangerous is how plausible the closed state looks. There’s a branch named after the ticket. There’s a commit on it with the right diff. If you glance at the JIRA dev panel you see linked commits and you feel done. The merge is the one step with no loud failure when it’s skipped, so it’s the step that quietly goes missing.

The second trap in the same session: two version conventions

While untangling this I tripped a related wire, and it’s worth naming because it’s the same class of error. To ship the hotfix I reached for git tag 3.349.0, a .0-style tag. Ben: “what the heck .0, we never use .0 seems like you dont underatsnd how hotfixes work.”

The cause is two version conventions sitting next to each other. JIRA’s fix-version field uses one shape (the ticket’s fix version was 3.349). The git hotfix tags use another: the latest tag on master was 3.348.10, so the next clean tag was 3.348.11, then 3.348.12, not a fresh 3.349.0. I’d read the JIRA fix-version 3.349 and turned it straight into a tag by bolting on .0. They look similar enough that you grab the wrong shape under load, and “similar enough to grab the wrong one” is the definition of a footgun. When your release identity lives in two places with two formats, the formats will collide, and they’ll collide at the worst moment, right when you’re trying to ship the thing that was supposed to already be shipped.

Verifying the deploy from the outside, not from my checkout

Once the merge actually happened, I refused to call it done from the local clone. The local checkout having the commit proves nothing about what customers hit. So I verified from production, against an endpoint that only exists after the deploy.

The fix shipped a new admin diagnostic endpoint. That endpoint is part of the change itself, which makes it a useful tell: if production serves the old signature, the deploy isn’t live yet. The first probe served the pre-fix signature. Not live. I waited and re-probed minutes later, and that time it returned all three carriers. Live. And because I was already looking at the real thing, I could also confirm the DB rows already carried the expected flag value, so the data migration was a verified no-op rather than an assumed one.

That outside-in probe is the part people skip. It’s tempting to verify against your own working tree because it’s right there and it’s fast. But your working tree is the one place guaranteed to have the fix. It can’t tell you whether the deploy ran, whether the right branch shipped, or whether prod is serving stale code from a binding you forgot about. The only honest verification hits the live URL from the outside, like a customer would.

The two rules I now gate on

The lesson isn’t “be more careful.” Being careful does not survive contact with four parallel agents and a 500-commit day. The lesson is to gate the status transition on a git fact, the same way I already gate a feature branch on the ticket first moving to CODING.

Rule one: a ticket cannot transition to STAGING/LIVE while its branch is unmerged. The transition has to assert a git reality, not a human intention. Before the status moves, check that the work branch is actually an ancestor of the release branch (git merge-base --is-ancestor <work> origin/master). If it isn’t, the transition is a lie and the tooling should refuse it. “Closed” should be impossible to type while the merge is missing.

Rule two: verify the live endpoint from the outside before you believe the deploy. Not the local checkout, which always has the fix. Probe the real production URL and confirm the new behavior is actually being served. If the change ships its own observable signature, like a new diagnostic endpoint or a version string, even better, because then “is it live” has a yes/no answer you can read off the wire instead of guessing from the deploy log.

Both rules are the same move pointed at two different surfaces: do not let a proxy signal stand in for the real one. A status field is a proxy for the branch state. A local checkout is a proxy for production. Each is convenient, each is usually right, and each will eventually be confidently wrong in exactly the way that costs you a customer-facing bug that your board swears is closed.

The takeaway

Closed, shipped, and live in your tracker are claims, not facts, until you verify them against branch state and against production. Gate the status transition on a git fact: the work branch must be merged into the release branch before the ticket is allowed to read LIVE. Then verify the live endpoint from the outside, not from the checkout that’s guaranteed to have your fix. The dangerous failure mode in fast solo dev isn’t wrong code. It’s a green ticket pointing at a branch that went nowhere.