Four Verdicts, Four Failures, and a Drive at Zero

On 2026-09-21 we found the drive that holds our work at zero bytes free. A cleanup sweep brought back 410MB.

That same morning our scheduled jobs were failing. We keep a small triage agent for that. It is a low-effort model call that reads the tail of a failed job’s log and writes one line saying why it failed. It exists so we do not open a terminal at 7 AM to learn whether a red mark means ignore it or act.

Four verdicts in ninety minutes

Between 7:19 and 8:49 that morning the agent wrote four verdicts, each for a different scheduled job.

At 7:19 it said the cause was unclear. A sweep had started at 04:41:40 and wrote no “done” line.

At 7:49 it said disk full. The remedy job crashed with an OSError, Errno 28, no space left on device, while writing a file named stderr.txt in remedy.py at line 882.

At 8:19 it said an expired login. The seat’s credential had lapsed, so the auth preflight refused to launch the runner with rc=3.

At 8:49 it said the cause was unclear again, because the runner’s own log did not show a failure.

What each verdict was and was not

The disk-full verdict was right, and the drive really was at zero. The login verdict was right too, on its own evidence. That was a separate fault from the disk, and clearing the disk would not have fixed it.

The two unclear verdicts are the open part. A log that starts a job and then stops with no end line is what a full disk can leave behind. Nothing on file shows that happened to those two jobs, so we cannot say the four failures were one problem, and we cannot say they were four.

What we can say is why no verdict connected them. The agent was asked about one failed job at a time. Each answer was correct for the question it was given. Nobody asked it whether the last four failures shared a cause.

The same error came back at 8:19 that evening. The triage agent reported Errno 28, no space left on device, for the health report job, which failed with rc=1 after 4 seconds.

What to check in yours

When several scheduled jobs fail in the same hour, look at free disk space before you read any single explanation. Then read the verdicts that said unclear and ask whether a full disk would produce the same silence.

dxdev LESSON · paste into your AI coding agent
LESSON: Check Free Disk Before You Read Any Single Failure
SOURCE: dxdev.com/blog/checkpoint-never-fired-nobody-checked
WHAT HAPPENED: A drive hit zero bytes free. In one morning a triage agent was asked about four failed scheduled jobs, one at a time. It said cause unclear, disk full, expired login, and cause unclear. A full disk can make a log stop with no end line, so a verdict of unclear may be the same fault as a verdict of disk full. No one question looked across the four.
THE RULE: When several scheduled jobs fail in the same hour, check shared resources first, starting with free disk space, before you accept any one job's explanation. Treat a log that simply stops as a possible full disk until you have ruled it out.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. The failure triage step receives the current free disk space of every drive the jobs write to, along with the failed job's log tail.
      2. When two or more jobs fail within a short window, a single check compares them and reports what they share, such as a drive, an account or a credential.
      3. A job that starts and writes no end line is reported as unfinished, not as success and not as an unexplained failure.
      4. Low disk space raises an alert before a write fails, not only after.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.