At 2:20 AM we changed our release runbook, because one release had exposed a drift we could not safely ignore.

The release itself went fine. The trouble was that our process could look at the repository and produce two different answers to one question: what comes next?

The runbook had been deriving the next release version from Git tags alone. That felt reasonable. A tag marks the commit that became a release, and it is easy to inspect, easy to sort, and easy to use as the starting point for the next version.

It is also incomplete. The merge trail held version information of its own, and that record had moved past what the tag check reported. The two histories had drifted. One of the affected releases belonged to a teammate, and the tag-only rule had no way to know it was returning a lower answer than the release history supported.

On paper that is a small failure. In a release process it is a bad one, because version derivation is supposed to be boring and deterministic. If the same repository answers with one value from the tags and another from the merge trail, the tool is no longer determining the version. It is picking which evidence to ignore. And the tag-only rule had already shipped a wrong number once before anyone traced it back to the runbook.

Two answers from one repository

The first useful observation was that this was not a single malformed release. It was a disagreement between two sources we already relied on.

The old flow asked one question:

What is the highest release tag?

The corrected flow asks two:

What is the highest release tag?
What is the highest version represented in the merge trail?

Then it takes the higher one before calculating the next release:

next_release = increment(max(highest_git_tag, highest_merge_trail_version))

That is the whole change in one line, and it changes the failure mode. A stale or missing tag can no longer silently pull the next release backward when the merge trail has already recorded a higher version. When the sources disagree, the runbook now reconciles them instead of trusting whichever one it queried first.

Tags stay in the flow as our explicit release markers. A merge-only check would just swap one blind spot for another, since the merge trail shows release work the tags may fail to reflect, and the tags show releases the merge trail may miss. Once we knew they could disagree, neither was authoritative alone.

We also left out a manual cleanup rule. Repairing the tag state around that release and telling people to be more careful would have fixed the symptom in one repository state and kept the same derivation bug for the next one.

A local tag proves nothing to anyone else

The second change was less glamorous. The runbook now verifies that the tag reached origin before it calls the release complete.

A locally created tag only shows that one working copy has it. If it never reached origin, release state is split again: the local machine believes the version exists, while the shared remote cannot confirm it.

So the revised sequence is ordered on purpose:

  1. Read the highest Git tag.
  2. Read the highest version in the merge trail.
  3. Take the higher of those two values and derive the next release version.
  4. Create and push the release tag.
  5. Verify the tag exists at origin.
  6. Only then treat the release as done.

That last check closes the loop on the evidence the next run will consume. The next operator should not have to guess whether a local action became shared repository state.

Sending it to the teammate who hit it first

We updated the written release flow and sent the change, along with a sync script, to the teammate whose release had surfaced one of the drifts. We designed the original tag-only derivation, and it produced a wrong answer for someone else’s release before it produced one for ours. That is a worse way to find out a rule is incomplete than catching it on our own work. Release tooling is shared infrastructure, so whoever sees the mismatch should be able to follow the same sequence, reproduce the decision, and check that the remote agrees.

The fix took 48 minutes. We added no new service and no database for version state. We made the existing evidence explicit, compared it before acting, and checked the remote after acting.

A release number is a conclusion drawn from repository history, and it can only be as good as the records it considered. Ours now considers both. If they disagree, it takes the higher version. If it creates a tag, it checks origin. If your next version comes from a single source such as tags alone, find out whether another record of the release history can disagree with it, because ours could.