We Shipped a Hotfix Down the Wrong Pipeline and Had to Cherry-Pick It Out

The ticket tracker showed the change still marked “in review.” Production was already serving it.

We’d built a review-request module for a major B2B software-review platform, meant to live on the post-checkout Thank You page, plus a rebuild of three existing domain modules into collapsible blocks to match a teammate’s page restyle, expanded by default. Small, self-contained front-end work. The kind of change that should sit in a review environment for a day before anyone touches prod. Instead it went straight through.

How the routing actually broke

The platform’s repo has a pipeline with two branch families that CI watches by name pattern. Anything matching staging deploys automatically to a shared review environment. Anything matching hotfix-* skips that environment entirely and deploys straight to production, because hotfixes are supposed to be small, urgent, and already reviewed elsewhere before they’re cut.

The branch for this work was named hotfix-checkout-thankyou, because “hotfix” was the closest existing label for “small, isolated, ticket-scoped change,” which is exactly what it was. It wasn’t urgent, and it hadn’t been reviewed. But the CI trigger doesn’t know the difference between “hotfix” as urgency and “hotfix” as branch-naming habit. It matches on the string. The branch name alone was enough to route the push straight to the production deploy job, no staging stop, no review gate.

I pushed it expecting the staging pipeline to pick it up so we could look at it live before any real users touched it. Instead the production deploy fired. The Thank You page on the live site had the new review-request module and the restyled domain blocks before either of us had looked at them outside a local browser.

What we saw and what it actually was

The first sign wasn’t an alert, it was the ticket state mismatch: the tracker said “in review,” the CI dashboard showed a completed production deploy job against hotfix-checkout-thankyou, timestamped minutes after the push. Pulling the deploy log confirmed it: the job had run against the production target, not staging. Nothing in the pipeline had asked for confirmation, because nothing about a hotfix push is supposed to need one.

Recovery was a cherry-pick, not a revert, because by the time we caught it there were already unrelated commits landing on top from other work. We identified the single commit sha for the Thank You page change, cherry-picked it onto a properly named branch cut from the last production tag, and let that one go through staging the normal way so we could actually look at it before it shipped again. The original hotfix-checkout-thankyou branch got deleted so it couldn’t fire the same trigger twice.

We reviewed the cherry-picked version live in staging, and it wasn’t done anyway. The domain blocks needed another pass on which sections should default open, and the branch is still sitting local and uncommitted with the current state written into the ticket’s dev notes rather than pushed anywhere, on purpose this time.

The actual fix

The bug wasn’t the deploy job, it was that “hotfix” meant two different things and the pipeline only understood one of them. We rewrote the branch-shape rule so the trigger doesn’t match on name alone anymore. A branch only qualifies for the direct-to-production path if its merge base is the current production tag. A branch cut from staging HEAD, no matter what it’s called, routes through the staging deploy and the review step, full stop. Calling something hotfix-anything no longer buys you a shortcut around review, only branching from a shipped tag does.

That’s a much narrower door than “contains the word hotfix,” and it means the failure mode we hit, a small unreviewed change riding a name it happened to share with the urgent-fix path, can’t happen from naming habits anymore. The rule now checks lineage, not vocabulary.

dxdev LESSON · paste into your AI coding agent
LESSON: Route Deployment Shortcuts By Branch Lineage, Not Names
SOURCE: dxdev.com/blog/2026-09-13_hotfix-wrong-pipeline-mistaken-route
WHAT HAPPENED: The work added a review-request module to a post-checkout Thank You page and rebuilt three domain modules as expanded collapsible blocks. Although it was ordinary review-bound work, its `hotfix-` branch name matched a CI rule that skipped staging and deployed straight to production. The team confirmed the mismatch in the deploy log, then cherry-picked the affected commit onto a correctly routed branch because unrelated commits had already landed on top of the original branch. After reviewing it in staging, they found further changes were still needed, and deleted the original branch to prevent another production trigger. They changed the pipeline so direct production deployment requires a branch whose merge base is the current production tag.
THE RULE: Deployment paths with different review requirements must be selected from verifiable branch ancestry, not a mutable naming convention. A production shortcut should require evidence that the branch originated from the current shipped version.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. A branch can enter the direct-to-production path only when its merge base equals the current production tag.
      2. Branches cut from `staging` HEAD route through staging and review regardless of their names.
      3. Renaming a normal development branch to include `hotfix` cannot bypass the staging deployment path.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.