When the Firewall Log Stopped at v14

Cloudflare had 11 live custom firewall rules. The rule ID index still said six.

That was worse than an out-of-date document. The version history in CF-WAF-RULES.md ended at v14, while the live ruleset had reached v27. We went in thinking we had a 13-version changelog backlog. We were actually fixing the missing comparison that had let the changelog drift in silence, on a document we had been treating as a reasonably current record of our own security posture. We do not know how many of those 13 versions we would have described confidently and wrongly if someone had asked us what our firewall rules did last week.

The original problem was already difficult. The rule set was versioned, but the useful account-level audit log could not be read with any of the available tokens. We could ask Cloudflare for every ruleset version and timestamp, and we could ask for a full snapshot of a numbered version. We could not depend on the dashboard to narrate the changes for us.

Then Claude supplied the first sharp warning about the investigation itself. Its initial collection job tried to write snapshots beneath /tmp/cf_versions and failed on the first v14 file. The directory it expected was not present in the environment it was using. It was a tiny filesystem mistake, but it interrupted the easy temptation to treat a version list as evidence. A list tells you when a ruleset changed. It does not tell you which rule, action, or expression changed.

We moved the collection into the project scratch area and fetched the full snapshots from v14 through v27. The data gave us 14 consecutive states to compare, rather than 13 lines to transcribe. We diffed each state against its predecessor and used the ticket comments only as a cross-check of the resulting entries.

The earliest diffs made the distinction tangible. Version 15 changed an ASN-focused rule from block to managed_challenge. Version 16 removed the truncated-Chrome branch from a user-agent expression, leaving the empty-user-agent match. Those were concrete rule changes with a prior state, a new state, and a timestamp.

Version numbers also exposed an indexing problem that ordinary prose had hidden. The log omitted v15 through v27, but the rule ID index was older still. It represented six rules at v11. The live ruleset had 11 at v27. We needed the chronology and the index to advance together, otherwise a future investigation could find a changelog entry without the ID required to trace the rule it described.

The crowded stretch was on August 21. Seven versions, v19 through v25, landed inside three hours. We had assumed the cluster would resolve into a single story once we found the surrounding ticket comments. That was another bad model. The version API can reconstruct custom-rule changes, while bot settings and AI-bot policies live outside this ruleset and are not versioned there. We could document the actual sequence of custom-rule edits. We could not credit the entire cluster to an unversioned setting just because the calendar was dense.

That limitation shaped the backfill. We added entries for versions 15 through 27 to CF-WAF-RULES.md in reverse chronological order, using the existing dated format. We refreshed the rule ID index at the same time. The work was documentation only. No firewall rule changed while we were closing the historical gap.

The second part was smaller in code and larger in consequence. We added a drift check to the weekly pre-release scan. It reads the live Cloudflare ruleset version, finds the newest Ruleset version N citation in CF-WAF-RULES.md, and warns when the document lags behind the API. That comparison is deliberately plain. It does not try to infer the motive for a rule change, classify the traffic, or synthesize a security verdict. It only asks whether the log has caught up to the rule set it claims to describe.

The first run after the merge reported that the gap had fallen from 13 versions behind to two. That remaining gap is why we trust the check more than the backfill. A one-time repair can make an old document look complete. A recurring version comparison makes its next omission visible.

The next weekly scan will compare the highest version in CF-WAF-RULES.md with Cloudflare’s live number. Two versions behind is now a warning, before it becomes 13.

Takeaway

A version list shows when a ruleset changed, not what changed, so a security change log can fall far behind without anyone noticing. Ask whoever maintains yours how it is compared against the live configuration. A recurring check that warns when the document lags the live version makes that gap visible before it grows.

dxdev LESSON · paste into your AI coding agent
LESSON: A Changelog With No Recurring Comparison Will Drift Until Something Forces an Audit
SOURCE: dxdev.com/blog/firewall-drift-detection-incremental-closure
WHAT HAPPENED: A markdown changelog tracking Cloudflare WAF firewall rule versions had fallen 13 versions behind the live ruleset, and its rule-ID index was worse, five rules out of sync. The fix pulled full snapshots for every missing version and diffed each against its predecessor to reconstruct actual rule changes (a block-to-managed_challenge change, a removed user-agent branch) rather than inferring from ticket comments alone, explicitly refusing to credit an unversioned setting (bot policies, which live outside the versioned ruleset) just because its changes happened to cluster on the same day. The more durable fix was a recurring drift check added to the weekly pre-release scan that compares the document's latest cited version against the live API version and warns when they disagree, so a 13-version gap becomes a 2-version warning instead.
THE RULE: A changelog or documentation file for a live, frequently-changing system (a firewall ruleset, an API version, a service registry) will drift indefinitely if nothing recurring compares it against the live source of truth; a one-time backfill repairs the current gap but the actual fix is a scheduled comparison that catches the next omission before it becomes 13 versions instead of 2.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any security-relevant changelog, runbook, or rule-tracking document with no recurring automated check comparing it against the live system it documents.
  2. Any documentation backfill or one-time repair for a drifted record with no follow-up mechanism to prevent the same drift from recurring.
  3. Any investigation that attributes a cluster of changes to a single cause based on timing alone, without confirming each individual change is actually covered by the same versioned/auditable source.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.