--- title: "A grading script gave 187 posts a clean bill of health" canonical: https://dxdev.com/blog/six-rules-for-a-blog-an-ai-writes/ datePublished: 2026-09-22 --- The score read `quality6: 6. avg: 84.` Every one of six gates was green. We pulled up the post underneath it to spot-check and it read like a press release: no ticket counts, no field names, no numbers at all. Somewhere in the pipeline, a rule meant to protect identity had eaten the whole post. That was the moment we stopped trusting our own grading script. Here is how it got built and how it broke. ## From one big read to a written rubric The first pass at auditing the blog was a single long session. We pointed an agent at all 72 published posts at the time and asked for a duplicate map, quality tiers (Flagship, Solid, Weak, Cut), and a recommended launch set. It worked. It caught real overlap, flagged one post (the April diagnostic writeup) as an unpublished duplicate to treat as cut, and named the sameness pattern that shows up after ten posts in a row. But it was one read, by one agent, on one day. Ask again next week and you'd get a different Weak/Solid split, because there was no written standard behind the tiers, just judgment. That's fine for a one-time audit. It's not fine as a permanent quality bar for a corpus that had grown to 187 posts with a drafts queue behind it. So we wrote the standard down. Six rules, each phrased as something a script could actually check: - no em dashes - at most one "X, not Y" construction per post - anonymize by substituting an equivalent specific rather than deleting it - open on a hard number or symptom - no templated headings - no closing line that just points at a companion post Six gates, pass or fail, per post. ## Wiring the gates into the pipeline The grading script runs every published post and every queued draft through repeated judge passes and writes one record per slug: `avg` (0-100, averaged across passes), `spread` (how much the passes disagreed), `gates_green` (how many of the six rules it cleared), `reframe` (keep or cut), and a one-line `suggestion`. All of it lands in a single `ranked.json`. A second, much smaller script does the triage. It reads `ranked.json`, sorts by `avg` descending, drops anything on a manually maintained drop list or already-done list, and takes the top 32 as `editorialTargets`, the posts worth a real second look this cycle. That second script is maybe fifteen lines. The bug was in the first one, in the gates themselves. ## The anonymize gate that deletion could pass The anonymize rule started out as "strip anything that could identify a person, company, or customer." Under that wording, deletion satisfied the gate. A post that used to describe a specific traffic pattern in numbers could have every number stripped, and the gate would score it green, because nothing was left in the post that could leak anything. Vacuously true. Zero signal, six gates green. We ran the full corpus through that version and got back a `ranked.json` with a healthy chunk of `gates_green: 6` posts. We fed the top scorers straight into the editorial-targets list without rereading them. That was the wrong turn, and it had a real cost: a batch of "keep" recommendations that were actually gutted. We only found out by manually rereading a sample afterward. One of them had a [specific number in its own URL slug](/blog/evidence-integrity-automated-pipeline/) and never used that number once in the body. The gate hadn't protected anything. It had deleted the article and called it compliant. ## Substitute, don't delete The fix was to change what the rule rewarded. Replace the identifying noun with an equivalent generic one, and keep every number, threshold, price, error string, command, and field name exactly as it was. A 63-unit organization instead of a named client. A tracker id instead of a real one. The count and the dollar figure stay untouched either way. We rewrote the gate check to fail a post that anonymized a proper noun but also dropped a number in the same paragraph. Real anonymization changes nouns, not digits. Re-running the corpus against the fixed gate demoted a real slice of the previous "keep" batch, correctly this time, because those posts had been passing on emptiness rather than on actual compliance. ## What the rubric has to survive A rule a script can satisfy by deleting the thing it's supposed to protect is a hole with a green checkmark on it. The useful question for any self-grading gate is whether it can be won by making the post worse. That question only shows up if you read the output instead of trusting the aggregate. `quality6: 6` is a summary. It is not a substitute for opening the file. The grading pass isn't a one-time audit anymore. It runs before anything ships, against the full 187-plus queue, and the six gates are the standard the drafts get held to, instead of a description of good writing we trust an agent to intuit fresh every time. When a quality score comes back all green, open a sample of the work underneath it before acting on the score, and ask whether the check can be passed by deleting the very details it is meant to protect.