---
title: "The 26 Minute Test Suite Number That Was Really 497 Seconds"
canonical: https://dxdev.com/blog/durable-artifacts-sdlc-playbook-standard/
datePublished: 2026-09-10
---
## Five of the six stages were already there

A newsletter pitched us an AI-native software development lifecycle playbook, the kind of pitch that promises a framework you're supposedly missing. We don't take a vendor claim like that on faith when we can check it against a real repo, so we pulled the playbook's actual write-up and read it against our own setup line by line. We already run five of the six stages it describes. Intent capture, planning, implementation, review, and shipping were all there, just under our own names.

The gap sat outside the stages, in durability. Every plan we made lived for exactly one session and then evaporated. The next session re-derived it from scratch, sometimes differently.

That cost is concrete. The review pass that runs before any code gets touched already produced a brief covering what would change, the order of work, and the open risks. But the brief was a chat reply rather than a file. If you came back six hours later, or a different session picked up the same ticket, there was nothing to read. You got a fresh interpretation of the same ticket instead of the decision that was actually made.

## The rename we shipped and had to undo

Our first pass at a fix took the playbook's structure and repackaged it as ours. We built a small tool that wrote a plan file with headers renamed to match our own vocabulary, something like Setup, Strategy, Guardrails, Proof instead of the playbook's own section names. It worked. It also missed the point.

It didn't survive review. The correction was plain: if you adopt an external standard, adopt it under its own names. A bespoke local rename is a different kind of forgetting, because now nobody outside this project can recognize the artifact as the thing it's supposed to be.

So we redid it. The tool now uses the playbook's own filename convention and its own four headers, enforced in that order, and it's wired into the review step so the brief survives the session that produced it. We also wrote a short verifying-your-work note into our house rules, so the expectation that plans get checked against something, and not just written, sits where every session actually reads it. Two of the playbook's other artifact types are still open. We're not claiming six of six.

## A guard test that passed without running

The same evening, closing out follow-ons on that work, we hit something worse than a missing feature: a guard test that had been passing without ever exercising the thing it guarded.

The guard exists to stop a hotfix-rules test suite from mutating a real, live clone of the platform's repo, which the test had been doing directly against disk. The test resolved its shell with Python's `shutil.which("bash")`. On this machine, that call returns WSL's bash, not git-bash. WSL bash can't resolve the repo paths the test hands it, so the pre-check step failed and the test's hook bailed out before it ever reached the guard assertion.

Then the test counted "hook bailed" as a pass, because the assertion was written to tolerate an early exit. The test had been green for a while, and it had never once run the check it existed to protect.

We fixed it two ways. We pinned the shell explicitly to git-bash instead of trusting `which` to find the right one. And we moved the test off the live platform clone entirely, onto a temp-directory fixture with an autouse guard that blocks any git mutation outside that directory. Now if the test can't run, it fails loudly instead of passing.

## Timing the suite instead of trusting the note

While we were in there we also [timed the test suite](/blog/test-passing-vacuously-bash-resolution/) instead of trusting the number written down for it. We'd been carrying 26 minutes as the full-suite time for a while, unverified, the kind of number nobody re-checks once it's in a doc. Actual full suite: 497 seconds, about eight minutes. That's still too slow to run before every commit, so we split out a fast subset, 2,859 tests in 71 seconds, that covers what actually changes day to day and can run without anyone thinking twice about the wait.

The plan that got thrown away every session, the guard test that passed without running, and the 26-minute figure share a shape. Each looked like it was doing its job, and none left evidence you could check without re-deriving it from memory. Plans, test results and timings only count once they exist as files someone can open and check.