Thirteen Scheduled Tasks Could Not Report a Failure

Thirteen of our scheduled wrappers could not report a failure, whatever the Python underneath returned. They hardcoded exit 0 or had no exit statement at all. The backup wrapper logged the real exit code and then never passed it on, so Task Scheduler’s LastTaskResult would read 0 even if the backup failed.

The wrapper that logged the truth and dropped it

The backup module’s main() returns 1 on a real dump or upload failure. The wrapper wrote that code into its log, then ended without an exit statement. The scheduler never saw it.

The gateway wrapper had the same gap. It recorded python returned $code, then stopped without exiting with it, so a daemon crash would be hidden from the scheduler and from the task health check. The listener wrapper never checked $LASTEXITCODE at all. A crash in any one channel’s sweep fell straight through to a default exit 0.

Why we read every return convention first

The audit covered every non-register wrapper except three that another session had uncommitted edits in. Each wrapper was checked against its module’s own return-code semantics before it was changed, because non-zero doesn’t mean the same thing everywhere.

Thirteen wrappers were broken. The fix commit’s message counted eleven, but its own file list and diff show thirteen, including the four CI alert runners. Six others were correct on purpose: their modules return 0 for success even with findings and non-zero only for a real failure, and those wrappers already documented that “findings, not failure” convention. We left them alone and added a matching comment to two more wrappers for the same reason, with no behavior change. Blindly converting every swallowed code into a propagated one would have traded silence for noise.

The fix

The fix was the same shape each time. Capture $LASTEXITCODE the moment the call returns, and exit with it explicitly. The listener runs several sweeps, and it now keeps any non-zero code from them as the wrapper’s exit code.

Keeping it fixed

We added a regression test that scans every wrapper for the swallow pattern, and proved it fires against a synthetic reproduction of the bug. A second test launches a real powershell.exe subprocess and checks that exit codes propagate, since the whole fix rests on that mechanism.

The backup wrapper wrote the real exit code to its log and then never returned it, so the scheduler never saw it.

dxdev LESSON · paste into your AI coding agent
LESSON: A Wrapper's Exit Code Must Propagate the Child Process's Real Result, Not Default to Success
SOURCE: dxdev.com/blog/eleven-tasks-reported-success-three-had-failed
WHAT HAPPENED: Scheduled tasks showed a successful result in the scheduler even when the underlying run had failed. The cause was a thin wrapper script between the scheduler and the real work: some wrappers hardcoded a success exit code regardless of what the underlying process returned, others had no explicit exit statement at all. An audit of every wrapper in the fleet found thirteen with the same bug and six others that looked similar but were already correct on purpose, returning a non-zero code for "here are findings" rather than "this crashed." Each broken wrapper was fixed to capture the child process's real exit code and return it explicitly, and a regression test now scans every wrapper for the exact swallow pattern that caused this.
THE RULE: Any wrapper, launcher, or orchestration layer sitting between a scheduler and the real work must explicitly capture and propagate the wrapped process's actual exit code; never assume a shell's default behavior will do this correctly, and never treat "non-zero" as uniformly meaning failure across different processes without checking each one's own return-code convention first.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any wrapper script (shell, PowerShell, batch, CI step) that calls a subprocess and does not explicitly capture and act on that subprocess's real exit code.
  2. Any scheduled job whose "healthy" signal (a scheduler's own success field, a dashboard's green check) has not been cross-checked against that job's own detailed log or output at least once recently.
  3. Any place a non-zero exit code is being treated as an automatic failure without first confirming what non-zero actually means for that specific process.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.