The Post That Never Printed Its Own Number 65

One of our published posts had “working-queue-grew-to-65” in its slug and never printed 65 anywhere in the body. Its own verify_notes admitted why: it had cut “ticket counts, field behavior, and internal workflow detail.” Six consecutive posts before it shipped with the identical three-heading skeleton, and two of them closed on the same sentence, word for word. That is what an automated content pipeline looks like when nobody audits its output for a while.

We wrote the rubric that told it to do this. The instruction to “redact internal architecture specifics” and “generalize” unsupported numbers was ours, added with anonymization in mind. It ran for six consecutive posts before we looked at what it had been publishing. This is a blog that runs on receipts, and our pipeline was swapping narrated conclusions in for the receipts we had told it to delete.

The pipeline is a daily sweep. It mines the work log and the JIRA timeline for blog seeds, hands each seed to a drafter, runs the draft through a review rubric, and promotes what passes. Two separate defects were producing the slop. Both were serious enough to file as root causes.

Root cause one: the rubric instructed deletion

The review rubric’s own text told the editor pass to “redact internal architecture specifics.” For a build-log post, architecture specifics are the whole point. The same rubric told it to replace a real business with “a client” and to “generalize” any number the draft couldn’t fully support. Generalizing a number does not anonymize it. It deletes the fact and leaves a vague sentence standing where a real one used to be.

The fix rewrites both the drafting prompt and the editor rubric around one distinction: substitution, never deletion. Names, company domains and ticket ids get replaced with a stand-in of the same shape instead of a bare “a client.” Numbers, thresholds, durations, error strings, commands, field names and tier names are explicitly listed as things that identify nobody, so they must survive verbatim. For a claim the sources can’t back, the rule flipped from “soften it” to “cut it entirely.” A missing sentence is honest. A hedged one wears the shape of a fact it can’t stand behind.

Root cause two: the drafter had nothing to draft from

The second defect was upstream. Nineteen backlog rows cited a repo @sha and resolved to zero sources. The git resolver only ever checked one repo root, and one of the archived repos lived on a different drive. When a resolver silently returns nothing, the drafter fills the gap with something plausible instead of refusing. That is where invented specifics come from: an empty context window getting padded.

Range citations failed the same way. One row cited a twelve-year commit range. git log --stat over that range emitted 8.8MB and blew the resolver’s 30-second timeout, so the row fell through exactly like the missing-repo rows. The fix summarizes instead of dumping: commit count, the oldest three commits in the range, the most recent forty subjects. That fits the token budget and is still true.

We also added resolution for two citation shapes the original resolver never covered. <repo> <TICKET> commits now resolves by grepping the log for the ticket key. Shorthand ticket runs like 7558/7560/7566/7570-7572 used to yield only the first number in the string, and now resolve in full.

The guard that would have stalled the cron

The identity scrub gate that runs after the drafter faced a version of the same choice. It checks eight literal banned tokens and is warn-only, so a leak can still ship. The obvious hardening is to make it fail closed. Before changing it, we measured what that would do: 71% of drafts in the queue would start failing. The daily cron would then produce nothing, and nobody would notice, because a silent empty output doesn’t page anybody the way a leak does. That trades a leak for a silent stall, which is a different failure mode wearing a safer costume.

What we built instead scrubs by substitution first. Each identifier gets a deterministic stand-in, with a private mapping file kept alongside the draft. The gate then re-scans the result and fails closed only if something survives the substitution pass. The pipeline keeps moving on the common case and stops hard only when the automated fix didn’t work.

Where it stands

Fifteen new tests pin the rubric and resolver changes. The full suite ran 416 passed against 7 pre-existing failures unrelated to this work. The audit turned up two gaps that aren’t closed yet, and we filed them as their own tickets instead of folding them into this fix. A separate fix for a stray false claim in the generator prompts is built and verified on a branch, still waiting to merge.

Treating “redact” and “delete” as synonyms was what was eating these posts. If you automate content or reporting, read what the pipeline actually publishes, and ask what each of its guards does when it fails. A guard that stalls output silently pages nobody.

dxdev LESSON · paste into your AI coding agent
LESSON: A Rubric That Says 'Generalize the Number' Is a Rubric That Deletes the Fact
SOURCE: dxdev.com/blog/evidence-integrity-automated-pipeline
WHAT HAPPENED: An automated content pipeline's own review rubric instructed the editor pass to "redact internal architecture specifics" and "generalize" numbers a draft couldn't fully support, treating anonymization and deletion as the same operation. For a technical build-log post, the specifics and numbers are the entire evidentiary content, so this rubric was silently hollowing out published posts, confirmed by a shipped post whose slug cited a number the body never printed. A second, separate defect (a git-log resolver that silently returned nothing for 19 backlog rows due to a wrong repo root and a timeout on long commit ranges) meant the drafter had no real source material for some posts and filled the gap with plausible-sounding invented specifics instead of refusing. The fix rewrote the rubric around one rule: substitute identifying names/domains/ticket-ids with equivalent-shaped stand-ins, but numbers, thresholds, durations, error strings, and field names must survive verbatim, and an unsupportable claim gets cut entirely rather than softened.
THE RULE: "Anonymize" and "delete" are not the same instruction, and a rubric or prompt that conflates them (redact specifics, generalize numbers, soften unsupported claims) will quietly strip the evidentiary content out of technical writing while looking, on the surface, like a responsible privacy pass; the fix is naming exactly what must be substituted (identity) versus what must survive verbatim (facts, numbers, mechanisms), and treating an unsupportable claim as something to cut, not hedge.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any content-review, anonymization, or redaction rubric/prompt whose instructions ("redact," "generalize," "soften") don't explicitly distinguish identity-substitution from fact-preservation.
  2. Any automated content pipeline with no periodic audit comparing several recent outputs against each other for structural or sentence-level duplication.
  3. Any data-resolution step (a git log lookup, an API call, a citation resolver) that can silently return empty or truncated results, with the consumer (a drafting or generation step) filling that gap with a plausible invention rather than refusing or flagging it.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.