The Third settings.json We Found Only Because a Docstring Named It

The diff command returned zero lines of output. Two settings.json files, one for our main work profile and one for the shared profile we run client work out of, five enforced hooks apiece, byte-for-byte identical.

An audit that started somewhere else

The audit began with something different: a pass over roughly 67 loose skill files that had piled up in .claude/skills/ with no index, no dedup, and a hand-maintained README that documented 12 of them and hadn’t been touched in over two weeks. While a subagent walked the skill tree, we took up the second half of the same question, hooks. Forty-one hook script files sat in one shared directory, wired into whatever settings.json each work profile pointed at.

We knew of two profiles. We compared them, got the empty diff, and started writing up “hooks are consistent across seats” as a finding.

The third file

Before we wrote the last line, we ran one more grep, this time for every settings.json on the machine instead of just the two we already knew about. It came back with three. The third wasn’t in our mental model. We found it because a hook script’s own docstring referenced a config path we didn’t recognize, pointing at a profile we had never accounted for: the one that runs the posting and automation work for this blog. It had been running the whole time and had never been compared to anything, because nobody had framed the question as “three profiles” instead of “two.”

Once we diffed all three, the picture changed. The third profile was missing three hooks that both of the other two enforce:

• estimate_gate, which blocks a dispatch from starting without a duration estimate on record
• reply_shape_gate, which checks that a generated reply matches the required output shape before it ships
• ask_shape_gate, which does the same check on the question-asking side of an agent exchange

It also carried two hooks the other two didn’t have at all: dispatch_budget, a spend guard on metered API calls, and hook_block_direct_browser_drive, which forces browser automation through a supervised path instead of a direct driver call.

How the drift happened

None of this was a deliberate decision. There’s no comment anywhere saying “this seat doesn’t need reply_shape_gate.” The two identical profiles match because one settings.json was copied into a second directory at some point and never diverged. The third profile was built separately, on its own timeline, for a narrower purpose, and picked up whatever hooks made sense for that purpose at the time. Nobody subtracted the enforcement hooks. They were never added, because the profile was never compared against the other two, and until this audit no artifact listed all three side by side.

The gap that mattered

The most important missing hook was reply_shape_gate. It catches a generated reply before it ships if it doesn’t match the required shape: no leading pleasantry, no meta-commentary about the draft, no sign-off tacked onto the end. It’s a small, boring, easy-to-forget guard, and it was absent on the one profile whose job is generating posts for public consumption. Nobody had decided that profile didn’t need it. The audit that would have caught the gap simply hadn’t been run, and the profile itself only surfaced by accident, from a docstring, mid-investigation.

Two matching profiles told us nothing about whether the system was well governed. They showed that we hadn’t looked hard enough to find the third. A diff on the pair we already knew about would have said everything was fine, and for those two it was. The finding came from noticing there were more things to compare than we had assumed, and going to count them instead of trusting the count we walked in with.

What we changed

The fix is a generated inventory that enumerates every seat’s hook set from the filesystem itself and is rebuilt on every audit. “How many profiles do we actually have” stops being a question anyone has to remember the answer to.

If someone tells you two configurations match, ask how they found the list of configurations. A safety check can be missing from one profile for no reason other than that nobody compared it to the others. Ask for an inventory built from the systems themselves, and have it rerun on every audit.

dxdev LESSON · paste into your AI coding agent
LESSON: Never Trust A Matched Pair Without First Confirming You Found All The Members
SOURCE: dxdev.com/blog/three-safety-configs-nobody-checked
WHAT HAPPENED: An audit of hook consistency across two known Claude Code work profiles found their settings.json files identical across five hooks each. Before finalizing that finding, a broader grep for all settings.json files on the machine surfaced a third profile, running the blog's own posting automation, that nobody had accounted for. Diffing all three showed the third profile was missing estimate_gate, reply_shape_gate, and ask_shape_gate, while carrying two hooks the others lacked. None of this was deliberate; the profile was built separately and never compared against the other two because no artifact listed all three side by side. The gap was especially notable because reply_shape_gate, which blocks improperly formatted output, was absent on exactly the profile generating public posts.
THE RULE: When auditing a set of things for consistency, first verify the enumeration of the set itself instead of trusting a remembered or assumed count, because a clean comparison only proves consistency among the members you already knew to check.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Does the audit process generate its list of targets (profiles, configs, seats) from the filesystem or environment itself, rather than from a hardcoded or remembered list?
      2. Is the target inventory rebuilt on every audit run instead of cached from a prior investigation?
      3. When a comparison returns 'identical' or 'no differences,' is there a separate check confirming the comparison set is complete before treating that result as a finding?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.