The publish log for a 23 hour 53 minute run showed zero errors, and the live post count went from 554 to 731. That is 177 posts. We opened every one of them on the live site anyway, because a log with no errors only tells you what the job believed it did.

What a clean log measures

The pipeline drafts a post, rewrites it, revises it, and pushes it. Each step exits 0 or it doesn’t. When every exit code is 0, the log says the job ran. It says nothing about whether the page a reader sees matches what the job thought it shipped.

So the spot check was a plain loop. Take a slug from the run, load it on the public site, compare it against what the log claimed. Two follow-up tickets came out of that, one against state tracking and one against authorization. I can show evidence for only one of them. The state tracking evidence is sitting in the ordinary daily logs. For the authorization one we don’t have a clean mechanism yet.

State tracking was already leaking in the daily logs

The nightly backfill job ran at 2:02 AM the same day and reported that nothing was published or held. Buried in its summary: six candidates turned out to be duplicates of posts that were already published, so it deleted them. A job that thinks it has six new posts to write, and finds out after the fact that they exist, has no reliable record of what it has already done. It recovered that time. Under a 177-post burst, that lookup gets many more chances to be wrong.

The nightly sweep tells the same story from another side. Its resync line that night read:

resync: 3 active ticket session(s), 3913 zombie thought folder(s), 4 stuck-active manifest(s)
needs-close: 219 still need a human /close

3,913 folders and 4 manifests all claiming to be live while nothing was running is what state drift looks like when it’s counted. None of it threw an error. Each record was internally consistent and only wrong against reality.

The same sweep’s checkpoint step logged this:

checkpoint: committed 0 repo(s): HELD (live session working there): notes 262 file(s)

That hold exists because of a bug we fixed the same day. The sweep used to commit other sessions’ in-progress work. Now it refuses to commit a repo while a live session is working in it. The blog session was the live one, holding 262 uncommitted files. It’s a small rule about who owns state, and it only exists because a log full of successful commits hid the fact that they were the wrong commits.

Why the run was on a different account

The night before, the blog jobs were blocked because the account they ran under was nearly out of usage. We moved the work onto the main account through Sunday and let it burn through the backlog in one long session. That kept the run alive. It also meant one run touched far more posts than the normal daily job, on a different account than the one the pipeline was built around. That is where authorization assumptions are most likely to break. We filed the ticket and did not pretend to have it solved.

Closing the gap between “exit 0” and “page is right”

It now flags any page that shows error text or is missing its main nav, and it only alerts on new failures, so a known-bad page doesn’t page us forever. A new 10-minute task runs a 13-page production check after every push, and the weekly crawl is back on.

Here is the state of that fix. A real push triggering the 13-page check has not been observed yet. A real chat alert has only been mock-tested. Its first weekly run will also list four crawl-artifact pages as new, and those will be noise. So the tool that exists to stop us trusting logs is itself verified only by a mock, and we’ll trust it after we watch it fire on a real push.

Until then, the rule we run by is simple. A log tells us the job finished, and only the live page tells us what got published.

What to check after a big automated publish

A clean log shows that a job finished, not that the live pages are right. After a large automated publish, load the live pages and compare them against what the log claimed. Then look at your production check: does it run after every push, and have its alerts ever fired on a real failure, or only in a mock?