One Malformed Seed, 43 Missing Posts, and a Misleading ENOENT

Five nights, one repeating line

Five nights running, the nightly blog drafter produced zero posts. Five nights running, the only signal we had was a CI triage bot repeating the same line every 45 minutes: publishPost returned ok:false (expected true), thrown by the publish gate test at line 125, with a secondary complaint at line 167 about an ENOENT reading the landing site’s content config file. It fired at 2:41 AM, again at 3:26, again at 4:11. Same cause string every time. It looked like infrastructure noise, the kind of thing that shows up in CI when a relative path resolves differently in the test runner than in production, and nobody escalated it because it had looked exactly like that before.

The ENOENT was real, but it was a test-harness artifact, and it was not the production failure. The test suite resolved the content config file relative to a working directory that didn’t match how the drafter actually ran at night. Fixing the path made the test pass and told us nothing about why the batch had been silent for five nights.

The regex we patched for nothing

Our first theory was the gate itself. One of the seeds queued that week had a dollar figure in its source material, something like a client’s monthly rate, and we assumed the leak-detection regex was choking on an ambiguous currency string, flagging it as a possible real invoice number and refusing to draft. We patched the regex to be stricter about what counted as a plausible dollar amount, redeployed, and waited for the next overnight run.

It ran. It still produced zero posts. That was a full night burned on a fix for a bug that wasn’t the blocker, on top of the four nights already lost, and the real failure sat unexamined while we were convinced we had found it.

Four failures in one run

We stopped trusting the CI triage summary and went straight to the drafter’s own run logs instead of the test suite’s interpretation of them. Four independent failures were stacked in the same nightly run, any one of which would have been enough to kill the batch.

The drafter refused part of its own seed payload. Seeds carry instruction blocks inline, and the model was reading an embedded directive as something closer to an injected command than a legitimate part of its own prompt, and declining to proceed. That pattern wasn’t new. Earlier the same night, a browser-manager role instance had refused a file read for a near-identical reason, treating a system-reminder-style block appended to its prompt as untrustworthy and stopping rather than acting on it. Same defensive instinct, different role, same effect: legitimate instructions read as suspicious and dropped.

A second seed tripped the leak gate, correctly this time. The detail that tripped it should have been masked before the seed ever reached the queue, instead of being caught at draft time.

A third seed was malformed, held in a bad state, and the batch runner had no per-seed isolation. One bad seed didn’t get skipped. It aborted the whole run, which is why all 43 posts queued for that stretch produced nothing. It was one failure with no containment around it, and it was never 43 separate failures.

A fourth: a collaborator’s name sat in an internal work log. That log had been pulled in as source material for six seeds, and the masking pass never touched it, because it only scanned the fields it expected names to live in, not every field a seed could source text from.

What changed

The batch runner now catches and skips a bad seed instead of aborting the run, so one malformed entry costs one post instead of the whole night. The masking pass scans seed-derived fields generically instead of an allowlist of expected locations, which is what should have caught the collaborator name the first time. The refusal pattern got a narrower instruction-block format that doesn’t read as an injected directive. And the leak gate stayed exactly as strict as it should have been on the one seed where it was right.

Once all four were fixed, that night’s run wrote 43 posts.

The silence itself

A job returning zero output for five straight nights generated no alert of its own. The only alert we had was a CI test complaining about a path resolution issue unrelated to the failure, repeating every 45 minutes for hours, indistinguishable from every other flaky-looking CI ping that gets triaged and ignored. Zero output and “tests are red” are different failure signatures, and we only had instrumentation for the second one. That gap has its own ticket now, and it’s next, because the next time something goes silent for five nights, we want to hear about it on night one.

Check that every automated job alerts on missing output, not only on failing tests. Then ask whether a silent batch at your shop would be noticed on night one.

dxdev LESSON · paste into your AI coding agent
LESSON: One Failure Should Cost One Item, Not the Whole Batch
SOURCE: dxdev.com/blog/three-failures-one-silence
WHAT HAPPENED: For five consecutive nights the blog drafter silently produced zero posts, with the sole visible signal being a recurring CI complaint about an unrelated path resolution bug. A first fix attempt, tightening the leak-detection regex around a dollar figure, cost another full night and turned out not to be the blocker. Going straight to the drafter's own run logs revealed four independent failures stacked in one run: a refusal to act on an embedded seed instruction read as an injected command, a correctly triggered leak gate on unmasked source material, a malformed seed that aborted the entire batch instead of being skipped, and a collaborator's name missed by a masking pass that only scanned expected fields. Fixing all four let that night's run produce 43 posts.
THE RULE: When a batch job can fail for multiple independent reasons, isolate each unit of work so one bad item can't abort the whole run, and instrument for silent zero-output failure specifically, since 'tests are red' and 'nothing happened' are different failure signatures that need separate alerts.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Does a single malformed or failing item in a batch abort the entire run, or is it caught and skipped so only that item is lost?
      2. Does anything alert specifically on zero output from a scheduled job, separate from and not dependent on existing test or CI failure alerts?
      3. Do content-masking or validation passes scan every field a record could source text from, or only an allowlist of expected fields?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.