Theme
Shipping safely
Releases, hotfixes, CI and tickets: knowing a change is really live and really done.
75 posts by Ben Downey, in two lanes: plain-language stories and the technical build log.
Everyday lane
Plain-language stories. No jargon. 9 posts.
Start here
Four Remote Records Decided the Release, Not the Local Folder
A release can be finished safely from a computer that never held the project files, if the right checks and human decisions stay in place.
Tags, stale branches and false signals 1
Hotfixes, release lanes and tracker truth 1
Making CI and deploy pipelines trustworthy 2
-
Why ScheduleTools260219.js Replaced ScheduleTools250805.js
A simple date in two file names let an older website deliver new page code without adding a large new system.
-
From 73 Percent to 11 Percent Without Touching the Deploy Tool
A file that appeared to vanish during a deploy was actually a warning that the machine had run out of breathing room.
Verifying staging, releases and customer paths 1
When passing tests still miss reality 1
Show 2 more chapters
Finishing a release means seeing it work 1
More in this theme 2
-
Where the Old Manage Admins Label Was Still Hiding
Two buttons still said "Manage Admins" after the page they led to had been renamed. One sat on the team details page. The other was in the chat add people box.
-
Livecheck Charge Kept Ahead of the DXdev Deploy
At 2:15 pm our AI agent skipped a code review finding because no live DXdev charge existed yet. At 2:19 pm the hub told it one did, and that it was being kept.
Technical lane
The build log: the logs, the code, the fix. 66 posts.
Start here
Why git describe named a version the release had already moved past
Closing a ticket exposed a silent assumption buried in our release workflow: git tags were the source of truth for hotfix version detection. They were not.
Tags, stale branches and false signals 13
-
Why git describe named a version the release had already moved past
Closing a ticket exposed a silent assumption buried in our release workflow: git tags were the source of truth for hotfix version detection. They were not.
-
The Tag We Created Locally Never Reached Origin
At 2:20 am, we changed our release runbook because one release had exposed a drift we could not safely ignore.
-
Duplicate Version Number After a 2 Hour 40 Minute Release
Two builds shared one version number. We found it after a release we had just verified, and the collision meant the release tag could no longer answer the one question we needed it to answer: which build is actually in production?
-
A Clean Audit of 211 Merges Missed Two Bad Release Tags
The audit was clean: 211 wrapper merges checked, zero missing a tag on origin. Ninety days of history, nothing untagged.
-
What Version 3.364 Still Owed After the Deploy Passed
The legacy release was live. Its closing pass did not exist.
-
Don't Trust the Green Deploy: Grep the Live File for Your Ticket Marker
A green deployment signal can confirm that a pipeline completed without proving a specific fix reached the serving artifact. Use a traceable artifact marker or a behavior-level check to verify the change that matters.
-
Where the ticket actually closed when the checkout was missing
A release workflow can use a remote control plane rather than a local checkout, but only when protected branches, approval evidence, provenance, deployment checks, and post-deployment verification remain explicit.
-
The Write Path That Returned Success and Skipped the Save
A long-idle branch carried a temporary write-suppression setting into release. The interface reported success while the intended changes did not occur, showing why stale branches need explicit source, behavior, and release-risk review.
-
Sixty write cases returned success:1 and not one touched a row
A single debug flag set to false in October 2023 rode a parked branch for two and a half years, merged in April 2026, and silently dry-ran every admin write in production. The postmortem is about how scaffolding debt outlives memory and what we changed to prevent it.
-
One Line of Config Left Over From a 2023 Branch
On April 28, 2026, we fixed a one-line setting that had been wrong since 2023: EDITDATABASE was still false.
-
When the dry_run branch sits above the is_ready check
A preview that skips a real precondition can produce a reassuring number and still fail at the moment of change. The solution is to share all decision logic and separate only the final side effect.
-
How to Triage an 880-Ticket Backlog Without Deleting Real Work
A triage tool queued a bug as fresh high-value work five days after the fix shipped under another ticket. One git log --grep would have caught it.
-
A `?prototype=1` Flag and 23 Commits: Prototyping a Redesign in Production Code
A query-string prototype flag can separate an internal presentation path from the default experience, but it does not isolate shared writes, totals, or payment contracts. Those seams need deliberate controls and verification.
Hotfixes, release lanes and tracker truth 15
-
Why a Four-Line Redirect Fix Shipped Before a 760-Line Refactor
A four-line redirect fix shipped as a hotfix while a 760-line refactor rode the preview branch. Route changes by blast radius, not diff size or the release calendar.
-
Why Our Hotfix Train Ran Without a Single Pull Request
A minor release and five hotfixes shipped across one day on a long-running legacy web application. Keeping each hotfix small made rollback scope easier to reason about under pressure.
-
Staging Had Twelve Players Per Group and Production Had 47
A feature deployed cleanly, then two data-shape defects surfaced in the first post-release window. The lesson is to treat production monitoring as part of completion, not an afterthought.
-
187 Commits Across Eleven Repos From Four Routine Customer Tickets
A normal customer-fix day exposed the propagation cost of carrying one application across eleven long-lived clone and release paths.
-
The Staff-Center Commits We Reverted Out of Develop
A seven-commit lag in local master looked like noise. Clearing it revealed in-flight modernization work already merged into develop, waiting to ship. Finding it was what mattered.
-
The 6-by-3 Isolation Matrix Became a Release Grid for Three Repos
A ten-minute review was supposed to be a ten-minute review. We opened a ticket to check on a staff-login follow-up, and it surfaced release-flow drift instead.
-
What a Pipeline Trigger Matching hotfix-* Actually Deployed
The ticket tracker showed the change still marked "in review." Production was already serving it.
-
The Hotfix That Lied: STAGING/LIVE in JIRA, Never Merged to Master
The ticket said STAGING/LIVE. It had a fix version, 3.349, and a real commit with a real diff.
-
JIRA Showed Done While Only One Sub-Branch Sat on origin/develop
A ticket marked Done had one of ten sub-branches on develop. Here is what we built after that discovery and why release workflow needs hard gates.
-
Two weeks of drift between a branch name and its ticket type
The branch was cut as a HOTFIX. By the time it merged, JIRA said staging task. Nobody had touched the type field on purpose.
-
Fifty-eight tickets in release 3.362 and sixteen had no Repo field
Sixteen of the 58 tickets in the 3.362 release had an empty Repo field, so "not in the diff" could mean three different things.
-
Replaying 385 recorded builds against a monitor that paged on one HTTP read
A Discord page at 8:12 pm said our web box and the Cloudflare edge were down. Prod was serving 800 to 1700 requests a minute straight through it.
-
Nine Minutes to Close a JIRA Ticket That Had No Branch
A ticket for a mail-server fix sat open in JIRA long after the fix was verified in production, because nothing in our close path could see the work.
-
Why Our Release Check Passed Minutes Before the Meeting
A report built to catch the board and the code disagreeing before a release meeting had a blind spot exactly where seven real disagreements were hiding.
-
Hotfixed but Unclosed Tickets After the 3.367 Release
Six 3.366.x hotfixes were live on master, customers were already seeing the fixes, and the ticket that tracked them was still open.
Safety checks that quietly fail 13
-
routing-260420.js Shipped Truncated Redirects and Nobody Knew Which Command Built It
A redirect bug turned out to be a silently truncated JS artifact with no documented compile step. The fix took an afternoon. The guardrail took one more hour.
-
Name260617e.js existed, yet three pages loaded Name.js instead
Six commits into a round of display fixes, three pages still pointed at /src/ source files instead of compiled bundles. A close-time guard now recompiles and checks before a ticket can close.
-
Terser, node -c, and the 18-Line Installer Behind Our Commit Gate
One broken minified JavaScript file got past us once. Our informal check was not a machine proving the result still parsed.
-
The shirt-size overflow that took down a whole registration import
A registration crash led to a length check we built into the wrong file entirely.
-
The Stale Bundle That Passed Every Green Check on develop
At 10:22 am on June 23, we marked the Customize Columns UI live on develop, then loaded a JavaScript bundle missing the opt-in changes that had just been merged.
-
Why a stored-procedure reorder collided with thirteen unrelated changes
A prod ship and a cache-header rollout forced a branch rule we had been working around.
-
Why Every Message From the New Links Arrived Marked Lowest Priority
A feature shipped clean. Eight minutes later, a second release went out to fix a default it had exposed, not caused.
-
A 231-Line Rewrite Showed Up as a Two-Line Bundle Diff
A staff tool for moving a customer account to a different configuration shipped in two patches and came back in one revert. The rebuild that held added nothing new to the page.
-
Two Hours Cleaning 30 Clones Found Two Fixes Marked Dead
The clone-cleanup job was supposed to be routine: 30 repos, fast-forward the stale base branches, delete dead leftover branches, close out the tickets sitting open against branches that no longer m...
-
The fromAdmin Early Return That Left Production Caching On
Four reviewers ran in parallel over a finished diff, and one of them flagged a preview render that would be written into the production page cache and served to real visitors.
-
The create call that succeeded and left 190 tickets unattributed
Five tickets had no Developer field, and every one of them looked fine the day it was created. That was the whole problem.
-
The Green Script That Passed Against V3 While Production Ran V4
The hub showed six sub-sessions still running. Five of them had closed their tickets over an hour earlier. Nothing was hung, nothing had crashed. The dashboard just hadn't noticed.
-
97 Stylesheet References and One Pasted Block of JavaScript
One filter button sat in the wrong place on mobile and in the right place on desktop, and the stylesheet that positioned it looked fine in the editor.
Making CI and deploy pipelines trustworthy 8
-
The Batch-Archive Change That Returned 200 While Leaving Orphaned Data
The console popped up first. The crawl script was shelling out to git with no window-hiding guard, and every so often a terminal would flash on screen mid-crawl.
-
When Your Pipeline Breaks on a New Repo, Blame Your Assumptions
A forwarded GitHub notification landed in our inbox at 9:46 am: semgrep had failed on our newest service repo, red X, no explanation beyond "1 check failed." It turned out to be our own deployment pipeline assuming a repo shape that had stopped being true.
-
One Private Dependency, Five Different Failures
Adding a private git dependency broke our blog deploy in five unrelated ways across two CI targets. SSH keys, git config overwrite semantics, GitHub's not-found lie, pnpm hoisting divergence, and a silent deploy you only learn about from email.
-
The Free GitHub Action Demanded a License Mid-Project, So We Called the Binary Directly
gitleaks-action@v2 asked for GITLEAKSLICENSE on a private repository after we had already wired the scanner into CI. The scanner had not changed. The wrapper around it had.
-
When deploy.ps1 Pulled Its Own Update Mid-Run and Kept Executing the Old Copy
Pushed a fix to a deploy script, watched it fail again at the exact same line. The script updates itself mid-run; the version in memory is always one run behind the version on disk.
-
What Our CI Said About Tests That Never Started
At 2:32 pm, a test command that had been broken for months finally started running in CI, and it immediately found the tests we had stopped seeing.
-
Five Committers Billed for a CodeQL Scan That Last Ran May 22
The invoice line item for GitHub Advanced Security did not match what the tool had done. The scan history for that repo said the last run was May 22.
-
The 5.28 Second Triage That Said Cause Unclear While the Site Stayed Down
At 3:41 pm the automated triage came back in 5.28 seconds, 76 tokens, thirteen cents: \"cause unclear: health check failed after deploy AND rollback also failed; underlying app failure not shown in ...
Verifying staging, releases and customer paths 7
-
A Staging Test Passed Two of Three Checks, and Both Passes Were False
A fix for duplicate customer emails broke a staff page on staging. The retest looked mostly green until we saw that the passing rows were written by production, not by the change.
-
Why the Domain Instructions Email Button Felt Slow
At 11:36 am on June 29, one click on Send Domain Instructions Email was taking the long way through our mail system.
-
Nine Days, 6,195 Rows, and a Ledger We Threw Away
We built a ledger to track which database scripts were applied, because verifying each one looked too expensive. Twenty minutes of testing showed it was cheap, and the ledger got thrown away.
-
The 38 Hour Session That Closed Without Opening the Deployed Page
Nineteen records, thirty days, one session that ran 38 hours and 27 minutes, and when it closed nobody had loaded the deployed page.
-
The Signed-In Customer Nobody Checked Before Launch
The new landing block showed every visitor the same pitch, including the ones who already had an account.
-
Voice, Tone, and Style Stopped Blocking Publishing on June 30
On June 30, we changed 40 lines in src/content.config.ts, added 33, removed 7, and stopped requiring voice, tone, and style before a post could publish.
-
The Bare Domain Returned a 404 Through the Mail Host
At 4:55 pm on July 21, a user could enter an invite code, write a decision into a Postgres-backed log, reload the page, and see the record still there.
Show 2 more chapters
When passing tests still miss reality 2
-
commission_balances Ran With Definer Rights in a Brand-New CRM
We put a card through the live payment account, the first payment it had ever taken.
-
We Picked 150,000 Password Hash Iterations and the Platform Said No
A prototype portal accepted any password. Turning it into a real login worked locally and failed on first live signup, because Cloudflare Workers caps PBKDF2 at 100,000 iterations.
More in this theme 8
-
Best-of-5 p95 rounds in test_units_store after the nightly flake fixes
A test_units_store p95 check with a 20 ms budget passed on the best of up to five rounds, so an intermittent slowdown passed it too.
-
Four Blog Tooling Paths Now Share One Function and 16057 Tests Passed
Four places in our blog tooling, the digest, social, lesson and dailybackfill, now get a post's URL id from one function, and that id carries no date.
-
506 open tickets, 429 candidates, and a verifier that would not agree
Eight of the 54 tickets our classifier called fixed or obsolete came back from an independent verifier with file:line evidence that the work was only partly done.
-
Why a pure insertion tripped the criss-cross check in our ship command
Our ship command refused a pure insertion when it hit a criss-cross conflict.
-
Why the FAILED Regex Went Through Three Forms
Four of the six nightly test failures got fixed, and the other two stayed red on purpose. They were a live-data alarm, and we left them failing.
-
A one-minute check found 16 orphan pages on master
At 7:39 one morning, a one-minute check told me our staff site had 16 pages that no menu linked to.
-
I Wrote the Same Mistake Into the Check Meant to Stop It
A fix was pushed and described as shipped while the process ran the old code. The check I wrote to catch that read the checkout on disk, and made the same mistake.
-
When Code Ships but the Database Doesn't: The Nine-Day Cost of Incomplete Releases
A season update on a test site took 39 seconds. After the fix it took 2.5.
Hitting one of these walls in your own codebase or your own machine? Start an assessment or talk to us.