--- title: "What Our CI Said About Tests That Never Started" canonical: https://dxdev.com/blog/graceful-test-degradation/ datePublished: 2026-06-23 --- At 2:32 PM, a test command that had been broken for months finally started running in CI, and it immediately found the tests we had stopped seeing. The cause was one character in the pnpm setup step. The setup action read the resulting configuration as a duplicate version pin and rejected it before the test runner ever had a chance to start. The test job existed. The workflow looked like it had a test step. The step that mattered was never executing. That cost us more than a green check. A [CI system can show a clean path](/blog/external-tool-state-unmonitored/) while an earlier setup failure keeps the thing you think it is validating from running at all. We had a test suite in the repository, a workflow that named the suite, and stale failures waiting behind a broken setup action. None of that added up to tested code. Once we removed the duplicate pin, the setup action accepted the workflow and the runner reached the tests for the first time in a long time. CI went red immediately. That was progress. For months we looked at a green badge and read it as "the tests pass." The accurate sentence was "the tests did not run." We don't know how many commits shipped in that window on the strength of a check that never checked anything. We would rather have caught that ourselves than had a broken setup step surface it for us. ## What the first red run showed The first run turned up two kinds of debt. Some tests were stale. They described behavior the code no longer had, so we corrected them. The failures worth a closer look were environment-dependent. A database-backed suite expected a database. Another suite expected the secrets store. The CI runner had neither. With the test step finally alive, the diagnostic path was short. We sorted the failures by where they started. The stale tests failed because their expected behavior was obsolete. The database and secrets-store suites failed at resource access, because their prerequisites did not exist in that runner. They showed no regression in application behavior. They showed that an integration suite cannot integrate with infrastructure nobody provided. Updating an obsolete assertion restores a useful test. Treating a missing database as an application failure turns an optional integration check into permanent noise. ## Making the prerequisite explicit The fix drew a line around the dependency instead of loosening what every test was allowed to assume. The database-dependent suite now checks whether its database resource is available. The secrets-store-dependent suite does the same for the secrets store. If the resource is absent, the suite skips itself. The ordinary tests keep running. In an environment where the database or secrets store is deliberately present, the dependent suite runs too. The sequence: 1. The workflow sets up pnpm without a duplicate version pin. 2. The test runner starts. 3. Tests with no external prerequisites execute normally. 4. A database- or secrets-store-dependent suite checks for its required resource. 5. It runs when the resource exists, or records a skip when it does not. A skip is not a pass, and that is why this works. A pass says the test executed and the expected behavior held. A skip says the test did not run because its declared environment was unavailable. Those are different facts, and the CI output has to keep them apart. If a resource is meant to be present and its connection fails, that still fails the suite. We are not swallowing operational errors. We are only refusing to claim that a test can validate an integration the runner was never configured to supply. ## Three directions that lost We looked at three other directions for this job, and each would have been worse. The first was provisioning every external dependency inside every CI run. That can be the right choice for a dedicated integration pipeline. As a first response here it would add moving parts to a check that otherwise needs to be fast and reliable. It would also make the test workflow responsible for infrastructure that most of the coverage never touches. The second was mocking the database and secrets store everywhere. That keeps the runner green, but it erases the distinction we needed. A mock tests behavior against a substitute. It cannot show that the real dependency is present or usable. For a resource-specific test, a universal fake turns an integration test into a unit test with a misleading name. The third was leaving the tests failing and treating them as environmental. That is how broken signals become background noise. Once a build contains failures nobody expects to fix, the failure that matters is much easier to miss. We had already lived through the larger version of this, where the test step itself was not running and the workflow never forced the issue. Self-skipping was the narrowest fix. It says exactly what the runner knows: this suite requires a resource, that resource is absent, and the suite has made no claim about behavior. ## Test suites have dependencies too We used to think of graceful degradation as an application concern. A feature should handle a missing service without crashing the whole request. The same rule applies to test infrastructure. A test suite is a program with dependencies, and it needs defined behavior when they are unavailable. Failing by default is sometimes correct, especially when the dependency is part of the environment the pipeline promises to build. Of the two worse outcomes, silently passing without executing does more damage than hanging, since a hang at least stops the pipeline and makes someone look. Skipping an explicitly resource-bound suite gives a third option that is more precise than either: it names the missing prerequisite instead of pretending the run was a pass or a failure. The one-character typo was the small defect. The larger one was letting CI imply coverage it was not producing. Fixing the setup action exposed that gap, and the truthful skip path keeps a successful test, an unavailable environment and a real failure distinct. The badge on the repository looks identical before and after. What changed is underneath it. A green check now means the pnpm setup succeeded, the test runner actually started, and every resource-bound suite reported its own status, run or skipped. Before, it could mean a setup failure that nobody had connected to the word "tests." If you take one thing from this, check that your test step executes, and that anything needing an outside resource reports a skip when that resource is missing. Then a green badge, a skip and a failure each mean one thing.