---
title: "A hash-chained manifest makes archive changes visible"
canonical: https://dxdev.com/blog/hash-chained-manifest-tamper-evident-85-lines/
datePublished: 2026-05-29
---
I wanted a low-cost way to notice unexpected changes in an archive: changes to the stored files, and changes to the record that describes them. A hash-chained JSONL manifest can provide useful **tamper evidence** for that purpose when it is paired with disciplined access control, retention, monitoring, and recovery practices.

It is important to be precise about the promise. A local hash chain can reveal many accidental or unauthorized changes, but it is not immutable storage and it is not proof against an actor who can rewrite every file, every manifest line, and the verifier's trusted reference. Higher-assurance uses need independently protected attestations or write-once storage, separated duties, key management, authorized change records, and an incident process. The small design below is a helpful integrity signal, not a complete security system.

## The setup

The archive in question is a pile of extracted artifacts. Each one gets scrubbed (PII redaction) and written to disk alongside a manifest. I needed tamper-evidence over two distinct things:

1. Did any **file** change after it was archived? (Someone edits a stored markdown.)
2. Did any **manifest line** change after it was written? (Someone edits the log itself to cover for the file edit.)

A naive answer covers only the first: store a sha256 of each file, re-hash on verify, compare. That catches file edits. It does nothing about someone editing the manifest, because the manifest is the thing you're trusting. If an attacker can rewrite a stored sha256 to match their edited file, your integrity check signs off on the tamper.

So the manifest needs its own integrity, and that's where the chain comes in.

## The data model

`_manifest.jsonl`, append-only, one JSON object per line. Each transcript extracts to a paired set of files (a canonical `.jsonl` the synthesizer reads, plus a rendered `.md` for humans), so each manifest line tracks both. A line:

```json
{
  "schema_version": "3",
  "transcript_id": "a1b2c3d4-...",
  "output_paths": {"jsonl": ".../2026-04-25-...jsonl", "md": ".../2026-04-25-...md"},
  "sha256":       {"jsonl": "<sha256 of stored .jsonl bytes>", "md": "<sha256 of stored .md bytes>"},
  "prev_sha256": "<sha256 of the previous line's exact bytes>",
  "scrubber_version": "<sha256(scrubber_code + scrubber_config)>",
  "findings_count": 41,
  "extracted_at": "2026-04-29T04:40:18Z"
}
```

The two fields doing the heavy lifting are `sha256` and `prev_sha256`.

`sha256` is a per-file content hash of each stored output on disk (one for the `.jsonl`, one for the `.md`). `prev_sha256` is the sha256 of the **previous manifest line's exact serialized bytes**. That second one is the chain. Line N carries the fingerprint of line N-1. Line N+1 carries the fingerprint of line N. Edit any earlier line and you change its serialized bytes, which means every `prev_sha256` from that point forward is now wrong. One edit, and the break propagates to the end of the file.

The first line carries `prev_sha256: null` (there's no previous line to fingerprint). Everything after chains off the real bytes of the line before it.

## The decision that makes or breaks it: deterministic serialization

A hash chain over lines is worthless if you can't reproduce the exact bytes that were hashed. `json.dumps` has options, and if any of them drift between write-time and verify-time, every `prev_sha256` mismatches and your verifier screams about tampering that never happened.

So pin the serialization and never touch it:

```python
def serialize(obj: dict) -> bytes:
    return json.dumps(obj, ensure_ascii=False).encode("utf-8")
```

Rules I locked down:

- `ensure_ascii=False`. Once you commit to it, you must keep it. Flipping it re-escapes every non-ASCII character and changes the bytes.
- **No `sort_keys`.** Counterintuitive, but the writer controls key order by construction (it builds the dict in a fixed order), so sorting buys nothing and adds one more knob that can drift. The hash is over bytes, not semantic content, so consistency is all that matters.
- No trailing whitespace, no indent, no trailing newline inside the hashed bytes. The `\n` separating JSONL lines is added when writing the file, not part of what gets hashed.

The `prev_sha256` for the next line is computed over those exact bytes:

```python
prev = hashlib.sha256(serialize(prev_line)).hexdigest()
```

If `serialize` is identical at write and verify, the chain reproduces perfectly. If it isn't, you find out immediately.

## Hash the stored output, not the raw input

This is the part people get backwards. It's tempting to hash the raw extracted content (the thing before scrubbing) because that's the "original." Don't. Hash the bytes you actually wrote to disk after scrubbing.

Two reasons:

1. **The verifier checks reality.** Verification re-reads the file on disk and re-hashes it. If you stored a hash of the pre-scrub input, the on-disk (post-scrub) file would never match, and you'd have to keep the raw input around forever just to verify. Hashing the stored output means verify is a pure function of what's on disk plus the manifest. Nothing else needed.

2. **`scrubber_version` catches redaction drift.** Each line stamps a `scrubber_version`, which is a hash of the scrubber's source code plus its config file. If the redaction rules change after extraction (you add a hard-deny term, say), the stored files were scrubbed under the old rules and the new `scrubber_version` won't match what a fresh run would produce. That tells you the archive predates a rule change, which is exactly the silent drift you want surfaced. You only get that signal if the hash is bound to the output the scrubber produced, not to raw input the scrubber never saw.

The general form: hash the artifact that will actually be read later, in the exact state it will be read. Keep any upstream data only when there is a defined, authorized purpose, an appropriate privacy basis, access controls, and a retention/deletion policy. Hashing an upstream representation without those controls creates both a weak verification story and unnecessary data exposure.

## The verifier (85 lines, stdlib only)

The verifier does exactly two checks and reports per-failure:

1. **Chain check.** Walk the lines in order. For each line, sha256 the previous line's exact stripped bytes (the raw line as read from disk, minus the trailing newline) and compare against this line's `prev_sha256`. The first line must carry `null`, since there is no previous line to fingerprint. Any mismatch means a line between here and the start was altered, and the break propagates to the end of the file.

2. **Content check.** For each line, walk both `output_paths` (`jsonl` and `md`), read each file, sha256 its bytes, and compare against the matching entry in the line's `sha256` object. A missing file or a hash mismatch fails the line.

Exit 0 if everything passes, exit 1 if anything fails, one line of output per failure. That's the whole contract. Sketch:

```python
prev_raw = None
for i, raw in enumerate(open(manifest_path, encoding="utf-8")):
    line = json.loads(raw)
    expected_prev = (None if prev_raw is None
                     else hashlib.sha256(prev_raw.rstrip("\n").encode("utf-8")).hexdigest())
    if line["prev_sha256"] != expected_prev:
        print(f"line {i}: prev_sha256 mismatch "
              f"recorded={line['prev_sha256']} expected={expected_prev}")
        failures += 1
    for kind, path in line["output_paths"].items():
        actual = hashlib.sha256(open(path, "rb").read()).hexdigest()
        if line["sha256"][kind] != actual:
            print(f"line {i}: {kind} sha mismatch for {os.path.basename(path)} "
                  f"recorded={line['sha256'][kind]} actual={actual}")
            failures += 1
    prev_raw = raw
sys.exit(1 if failures else 0)
```

Note the failure messages are deliberately distinct: `md sha mismatch` (or `jsonl sha mismatch`) for a file edit, `prev_sha256 mismatch` for a manifest edit. When the verifier trips, the message tells you which surface got touched.

## Two live attacks

I didn't trust it until I tried to break it. Two attacks, both surgical.

**Attack 1: inject one byte into a file.** Open a stored markdown, write a single byte at offset 100, save. Run the verifier. It reports `md sha mismatch` on that file's line and exits 1. One byte, in a file that's tens of kilobytes, and the content check catches it. That's just sha256 doing its job, and it's the easy half.

**Attack 2: edit a number in the manifest itself.** This is the interesting one, because it's the attack the naive design misses. I changed a `findings_count` field on one manifest line from 375 to 9999, the edit you'd make to hide that you'd quietly stripped content. The content hashes of the files were untouched, so a file-only check would pass. But the chain caught it: editing that line changed its serialized bytes, so the **next** line's `prev_sha256` no longer matched. The verifier reported `prev_sha256 mismatch` on the following line and exited 1.

That is the value of the chain in this threat model: an edited line may look internally consistent, but a later record exposes that its byte sequence changed. The result is evidence for investigation, not an automatic conclusion about intent or attribution.

Restore both files to their original bytes, re-run, exit 0. The tamper-evidence is symmetric: it doesn't matter whether you edit a file or edit the log, because the log's integrity is anchored to the log's own byte sequence and the files' integrity is anchored in the log.

## The one operational wrinkle: legitimate rewrites

Append-only is the happy path, but a legitimate schema change or re-scrub may require a controlled rewrite. Treat that as a new, authorized attestation: require a documented reason, appropriate approval, a tested plan, and preservation of the prior manifest where policy permits. Write the replacement to a temporary file, validate it, then rename it atomically on the same filesystem so an interruption does not leave a torn record. Record who performed the change, what inputs and rules changed, how the result was verified, and how to recover if the migration is wrong. A re-chain is not a repair-in-place or a reason to erase prior evidence; it is a governed lifecycle event.

## Related

- [Your Committed Output Is a Free Golden Test](committed-output-is-a-free-golden-test): using stored output as an integrity baseline, the same principle as hashing the archived files
- [The Most Restricted Tool in My AI Pipeline Is the One That Writes the Output](synthesizer-is-your-exfiltration-path-blast-radius): containing what the pipeline can exfiltrate, the threat model the manifest guards against
- [Why Publication-Time PII Redaction Quietly Re-Leaks Forever](redact-at-the-entrance-not-the-exit-recursion): redacting at the right pipeline boundary so there is nothing to tamper with downstream
- [rtk Wasn't Broken, It Was Unreachable: How I Found 366 Silent Failures in My AI Agent Sessions](audit-ai-agent-transcripts-like-flaky-test-suite): auditing AI pipeline outputs for silent failures, the same discipline as running the verifier