Why a green drift check on v27 hid a crawler bypass at v28

The drift check ran on the day it was supposed to prove itself and reported four versions of drift: live ruleset v31, change log v27. The number was correct. The docs were still wrong in ways it never looked at.

Where the check came from

We keep the firewall’s custom ruleset in a markdown file with a change log at the bottom. The log had fallen behind, so we reconstructed versions 15 through 27 by diffing consecutive snapshots from the Cloudflare rulesets version API. We also refreshed the rule ID index in the same file, which still said 6 rules at v11 when the live ruleset had 11 rules at v27. Then we added a drift check to the weekly pre-release scan so the log could not fall behind again.

The check took the newest Ruleset version N citation in the change log and compared it to the live version. That was the whole check.

What it never read

Two other summaries sat beside the change log. The rule ID index in the same file said v27, 11 rules, against a live v31, 13 rules. A second doc, the firewall notes kept alongside the agent instructions, said 8 rules, ruleset version 18. It also still described the known-good-automation skip as phase-only.

That claim stopped being true in v25. It is the exact misunderstanding that took a payment webhook down earlier, so the stale doc was teaching the mechanism that had already caused an outage. A doc can cite the right version and still teach the wrong mechanism.

The docs also knew nothing about the crawler leak fix. The bypass rule skips on cf.client.bot, which is true of every verified crawler, so ClaudeBot and Amazonbot were waved past before the training-crawler policy was consulted. The fix was a new block rule placed above the skip, and it went in at v28. Our change log stopped at v27. The skips are no longer contiguous either, because rule 2 blocks AI training crawlers and must sit above the verified-bot skip.

The backfill got v28 to v31 wrong

Our first pass at the newer versions had errors of its own. We dated v28 and v29 to 08-27 and attributed v30, a .php exclusion added to the skip rule, to the crawler ticket. The version API says v28 and v29 both landed 2026-08-25 and v30 landed 2026-08-26, and the crawler ticket’s own branch never mentions v30 at all.

That cost us twice. The crawler ticket’s branch carried its own unmerged v28 and v29 entries, and merging it collided with our rewrite of the same file. This is the hazard the v29 entry itself warns about, two sessions editing this ruleset in the same hour, arriving inside the doc that warns about it. We kept our structure and took the branch’s entries verbatim, because they carried what our reconstruction lacked: 431 served and 0 blocked, 438 skip events, and the point that lowercasing the match would block 12,406 wanted Claude-SearchBot requests a day.

Then we had to write a correction on the crawler ticket. v30 now reads “no ticket record”, with the nearest open question named as context and marked as a reading, not a record.

Generating the index

So the index is generated. A local command builds it from the live ruleset and writes it between markers in the file. Its --check mode asserts that the committed block still equals what the API returns. That single equality covers rule count, ids, order, actions, enabled state and skip scope together, so none of them needs a check of its own.

Prose cannot be generated, so it gets a different treatment. Counts in prose, in both docs, are now linted against the live ruleset. The lint applies only above the change log, since a change log records past states by design and an old number there is correct. Any skip that is not ruleset:current is reported on sight. The drift command now counts each disagreement as an anomaly instead of printing one advisory line.

We rewrote the intent table in the firewall notes to the real 13 rules and labelled it as intent rather than inventory. A short section now says which of the three homes owns which fact. Nothing had said so before, which is why they disagreed.

Verification

The command now reports “docs agree with the live ruleset (v31, 13 rules)”. 1,661 tests pass, 7 of them new. They pin the failure modes from that day:

• a stale index while the change log is current
• the staff doc being checked at all
• the generated block not reporting its own count
• a phases-only skip being flagged

We verified the merge by reading both files back from origin, from the remote copy and never the local tree.

One question stays open. A second opinion argued the two docs should merge, since “private” means little once both sit in a staff-visible repo. That is a structural call in the middle of a knowledge-base migration, so we left it alone.

A check that compares one number will pass on the strength of a current change log while the summaries beside it go stale. Ours did exactly that, and the crawler block at v28 appeared in none of the docs. Generate what the live system can tell you, assert the committed copy matches it, and lint the counts in the prose that remains.

dxdev LESSON · paste into your AI coding agent
LESSON: A Drift Check Must Compare Everything A Reader Relies On Against The Live Source, And Generate What It Cannot Compare
SOURCE: dxdev.com/blog/self-generating-firewall-rules
WHAT HAPPENED: After a documentation backfill, I added a drift check that compared the newest version cited in the change log to the live ruleset version. It reported four versions behind, which was true, but it was the only thing it inspected. A rule ID index in the same file and a separate staff doc both carried stale counts and versions, and the staff doc still taught a skip mechanism that had changed in v25. My own backfill of v28 to v31 also had wrong dates and a wrong ticket attribution, and merging the crawler ticket's branch collided with my rewrite of the same file. I fixed this by generating the index from the live API with a check mode that asserts the committed block equals the API output. Counts in prose are now linted against the live ruleset above the change log, and each disagreement counts as an anomaly.
THE RULE: A drift check should compare every summary a reader might rely on to the live source. Anything that cannot be compared should be generated from that source instead of maintained by hand.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Every doc or section that states a count, version, or mechanism about a system is covered by an automated comparison against the live source, not only the newest change log entry.
      2. Any derived index or inventory is generated from the live source, and a check mode fails when the committed copy differs from what the source returns.
      3. Each fact has one designated owning location, and prose that restates numbers is linted against the live source, with historical logs excluded because stale values there are correct.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.