calendar_day Raised an Exception Every Night and Nobody Heard

The nightly run at 2 AM exited with code zero four nights in a row, and on every one of those nights the calendar_day step had raised an exception and produced nothing.

The pipeline turns our work log into a searchable system. It picks up transcripts, enriches them with thread metadata, stamps each session with an area, builds a calendar index, and writes the result back into our archive so later sessions can reason about earlier ones. It runs as a Windows scheduled task that calls a PowerShell script that calls a chain of Python steps.

We built a health dashboard partly so this pipeline could not rot in silence. The command prints a one-screen invariants dashboard: queue depth, enrichment lag, provider mix, remote host reachability, and the freshness of the nightly cron. The idea was to glance at it and see when something had stopped working before it stopped mattering.

It took four days for us to notice anyway.

the morning audit

The trigger was small. The calendar index for a recent day looked thin in the dashboard, so we went to check why. We expected a one-line answer. We ran the pipeline manually with verbose output and watched it walk the steps. calendar_day raised an exception, the outer driver caught it, logged a one-line summary into .scratch/cron-refresh.log, and carried on. The script finished with exit code zero.

Every other step had run. The calendar step had not. That was what had been happening each night: the cron reported success while one step returned nothing.

The dashboard read the script as healthy because the script said it was healthy. The script said so because the driver had decided, when this code was first written a year ago, that one step failing should not bring down the whole pipeline. That was reasonable at the time. The driver caught the exception, logged it, and kept going, so the exit code stayed clean. The dashboard watched the exit code.

So the dashboard was monitoring for cron failures, and the cron had not failed. Failures inside the cron job were a different thing, and we had not realized those were the ones we cared about.

why the partial-success default is so dangerous

We have written this pattern several times, and it usually looks reasonable. Each step is independent. If one blows up, the others can still do useful work. Returning a non-zero exit code for a partial failure would trigger downstream alerts, restarts and retries, all the heavy machinery that “the job failed” sets off. For a partial result you don’t want that machinery. You want the steps that worked recorded, the step that failed flagged, and tomorrow to be another chance.

The catch is that “flagged” has to mean something a watcher can see. If “flagged” is a single line in a log file that nobody reads, then flagged is the same as lost.

calendar_day had been flagged every night. The log file held four flags. Its only consumer was a human reading it, and the first reading happened on day five.

the five-round fix

We worked the cleanup in five small commits across the day. Each looked obvious in hindsight, and each was something we had not done a week earlier.

The first changed the driver to write the full traceback for any failing step into cron-refresh.log, instead of only the first line. The first line of an exception is usually the least informative part. It is the framework’s wrapper, and the actual cause sits several lines down. We had been throwing that away every night.

The second added a failed_steps counter that the driver propagates to the outer cron summary at the end of the run. The summary now ends with a POST_ENRICH_FAILED_STEPS line, so a grep finds every failure in the log without reading the whole file.

The third made the exit code honest. The script now returns zero only when every step finished clean. If any step raised, it exits non-zero. The cost is that any single failing step now lights up the alerting that exists at the cron layer. The benefit is that “exit code zero” means what it says.

The fourth was about the underlying step. calendar_day had been raising on a specific kind of malformed session that started showing up four days ago, after an earlier change in the area-stamp logic widened the set of sessions it accepted. The fix to the step itself was small. Finding out we needed it took the first three commits.

The fifth added a Cron failures (last 24h) section to the dashboard. It parses the new structured failure blocks in cron-refresh.log, counts them by step name, and surfaces the five newest with timestamp, label and exit code. The dashboard now reads the file we had been writing the truth into, instead of trusting an exit code that had been hiding it.

the part that bothers us

The whole sequence took less than a day to fix. Reading the right log file would have taken five minutes on any night during the gap. We didn’t, because we trusted the dashboard.

We built the dashboard to be the thing we trust, which is the entire point of a dashboard. If it is wrong, we are badly placed to notice, because by definition we don’t look behind it until something breaks visibly.

The shape of the bug: the watcher was watching a proxy for the truth, the proxy stopped correlating with the truth, and the only feedback loop that would have caught it was a manual check we had explicitly stopped doing because the dashboard existed.

The only defense we see is paranoia about what each instrument actually measures. The dashboard measured exit codes, and we had decided exit codes were a meaningful signal. That decision was wrong for this pipeline from day one, and it took four days of silent rot to show it.

what we are sitting with

We built the right monitoring for the wrong question. We asked “did the cron fail?” and the pipeline answered no every night. The question we needed was “did every step in the cron do what it was supposed to do?”

The fix we shipped changes the answer to the original question: “did the cron fail” now returns yes when any step inside it raises. It is still the wrong question in one sense, because it treats every partial failure as a full failure, and that will overfire alerts in cases where one step should be allowed to fail without the rest of the pipeline noticing.

We don’t have a clean answer for that yet. The working answer for tonight is that any failure is loud and we will see it. The harder version, where each step’s outcome is independently visible to the dashboard, is still in front of us. The right shape is probably per-step success records that the dashboard reads directly, with no derived exit code at all. We don’t need to solve that this week.

The proof that this changed is in the log. Four nights of silent failure used to produce four one-line entries nobody read. Now a failing step writes a full traceback into cron-refresh.log, adds itself to the POST_ENRICH_FAILED_STEPS line at the end of the run, and shows up in the dashboard’s new failures section with a timestamp, a step name and an exit code. If calendar_day breaks again tomorrow, a thin calendar index won’t be what tips us off. The failed-steps line will already say so.

what to check on your own jobs

For any scheduled job, find out what its health signal actually measures. Every dashboard panel is a claim about the system underneath, and each claim needs checking now and then against the system itself, rather than only against the panel.

dxdev LESSON · paste into your AI coding agent
LESSON: A pipeline step that swallows its own exception to keep the job running needs its own visible failure signal
SOURCE: dxdev.com/blog/cron-rc-zero-isnt-success
WHAT HAPPENED: A nightly cron pipeline's outer driver caught exceptions from individual steps, logged a one-line summary, and continued so the rest of the pipeline could still run, returning exit code zero even though one specific step had silently failed for four straight nights. The health dashboard monitored the cron's exit code as its freshness signal, so it reported healthy the entire time, and the only consumer of the actual failure log was a human who happened to open the file, which didn't happen until day five. The fix, five commits, logged full tracebacks instead of only the exception's first line, added a failed-steps counter surfaced in the cron summary, made the exit code honest by returning non-zero if any step failed, fixed the actual failing step, and added a dashboard section that reads the structured failure log directly instead of trusting the exit code.
THE RULE: When a pipeline step's failure is intentionally caught and logged rather than propagated, a reasonable choice to avoid one bad step killing the whole run, the thing that catches it has to be a signal an automated watcher actually reads, not a log line only a human might open. A monitoring dashboard is a claim about what the underlying system is doing, and that claim needs to be checked against the system directly on occasion, not trusted indefinitely.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any pipeline step wrapped in a try/except that logs and continues, where the resulting exit code does not reflect that a partial failure occurred.
  2. Any health or monitoring dashboard whose freshness or success signal is a derived proxy, such as an exit code, rather than a direct per-step success record, with no periodic manual check confirming the proxy still correlates with reality.
  3. Any exception-logging code that captures only the first line of an error rather than the full traceback, discarding the most diagnostically useful part of the failure.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.