---
title: "Four Verdicts, Four Failures, and a Drive at Zero"
canonical: https://dxdev.com/blog/checkpoint-never-fired-nobody-checked/
datePublished: 2026-09-21
---
On 2026-09-21 we found the drive that holds our work at zero bytes free. A cleanup sweep brought back 410MB.

That same morning our [scheduled jobs were failing](/blog/eleven-tasks-reported-success-three-had-failed/). We keep a small [triage agent](/blog/blog-pipeline-end-to-end/) for that. It is a low-effort model call that reads the tail of a failed job's log and writes one line saying why it failed. It exists so we do not open a terminal at 7 AM to learn whether a red mark means ignore it or act.

## Four verdicts in ninety minutes

Between 7:19 and 8:49 that morning the agent wrote four verdicts, each for a different scheduled job.

At 7:19 it said the cause was unclear. A sweep had started at 04:41:40 and wrote no "done" line.

At 7:49 it said disk full. The remedy job crashed with an OSError, Errno 28, no space left on device, while writing a file named stderr.txt in remedy.py at line 882.

At 8:19 it said an expired login. The seat's credential had lapsed, so the auth preflight refused to launch the runner with rc=3.

At 8:49 it said the cause was unclear again, because the runner's own log did not show a failure.

## What each verdict was and was not

The disk-full verdict was right, and the drive really was at zero. The login verdict was right too, on its own evidence. That was a separate fault from the disk, and clearing the disk would not have fixed it.

The two unclear verdicts are the open part. A log that starts a job and then stops with no end line is what a full disk can leave behind. Nothing on file shows that happened to those two jobs, so we cannot say the four failures were one problem, and we cannot say they were four.

What we can say is why no verdict connected them. The agent was asked about one failed job at a time. Each answer was correct for the question it was given. Nobody asked it whether the last four failures shared a cause.

The same error came back at 8:19 that evening. The triage agent reported Errno 28, no space left on device, for the health report job, which failed with rc=1 after 4 seconds.

## What to check in yours

When several scheduled jobs fail in the same hour, look at free disk space before you read any single explanation. Then read the verdicts that said unclear and ask whether a full disk would produce the same silence.