The Weekly Restic Backup Had Only One Snapshot in S3

One snapshot. Dated April 2. Tagged initial.

The repo was the one we’d been running our AI operations out of, a thing called hive. We froze it in place as hive-archive and split its live concerns across four new sibling repos. We expected a slow afternoon of rewriting hardcoded paths inside the old repo. Once we started cleaning up the runtime around it, we found a field of dead automation we’d been trusting to be alive.

What hive was

Hive was a single repository that tried to hold everything our AI setup needed. The list that matters:

• A multi-agent dispatch engine with a FastAPI dashboard.
• A 40-table Postgres database of task state and logs.
• Our private memory store, blog drafts, research docs, and plan records, all in-tree with the engine.
• Deployment configs for user systemd services, cron jobs, and a Postgres container in Docker.

In good weeks, everything worked. Claude Code opened from that directory, auto-memory populated, every tool knew every path. In less-good weeks, any content commit could break the engine, and any engine debug could strand a blog draft. The coordination tax on every change climbed slowly enough that we didn’t notice until we did.

This wasn’t our first time hitting this wall. A handful of earlier single-repo projects, plus a side-project web app, plus hive. Each began in a single repo and each ended in the same tangle: concerns fuse inside a monorepo, isolation becomes impossible, and the exit ramp is a rewrite.

This time we didn’t want a seventh rewrite. We wanted to understand why the same shape kept producing the same failure, and to separate the concerns before they fused again.

Five repos, one operator surface

The new topology is five repos at ~/projects/:

• ops-repo: thin orchestrator. Scripts, conventions, routing. No product code. This is the directory Claude Code opens from, always.
• logs-repo: high-velocity private content. Memory, plan drafts, daily learnings.
• content-repo: public-bound content. Sanitized docs and feeds consumed by the live site.
• source-archive: raw, immutable source material. ChatGPT exports, transcripts. Things we might want to re-derive from but never edit.
• hive-archive: the frozen former hive. Preserved, never modified.

Any developer can tell you to separate concerns. What shaped this split is that Claude Code’s auto-memory lives in a directory keyed to the cwd it launches from. Launch from ~/projects/foo and you get memory at ~/.claude/projects/-home-username-projects-foo/. Launch from somewhere else and you get different memory, different accumulated context, a different mental model.

So the commitment is to always launch Claude Code from the same directory, even when operating on N other repos. That directory becomes the operator surface. The other repos are data, touched via absolute paths. One CLAUDE.md, one memory store, one mental model. ops-repo is the cwd.

One structural detail inside the split: the migration plan had nine steps, and step two, shipping a static blog content API from a public repo to our landing page, was pre-committed as a hard gate. Nothing in steps four through nine was allowed to start until step two was live. Our specs had been running ahead of our implementations for months, and the blog API was deliberately chosen as a small, well-specified, achievable ship. It landed: a Python generator reading markdown frontmatter, writing a JSON feed in a public repo, fetched by an Astro content loader on the landing site. Technically small. As a gate, load-bearing.

The runtime was half dead

Step six of the plan was rewriting the 43 runtime files inside hive that hardcoded the old path. That worked. What the plan didn’t catch was runtime automation that hardcoded the old path but lived outside the repo: in our crontab, in ~/.config/systemd/user/, in Docker.

When we went to clean those up, we expected to find a handful of stale references.

We found six crontab entries (verbatim copy saved to ops-repo/backups/2026-04-23/crontab.txt before crontab -r cleared them), seven systemd units (five of them enabled at boot, so the next reboot would have produced a cascade of “failed to start” states), two orphan MCP processes, and a retired Postgres container. Normal cleanup. Then we started actually reading what the dead crons had been trying to do.

The weekly restic backup cron, scheduled Sunday 4 a.m., was supposed to push a snapshot of system state to S3 every week. We checked what was in the bucket. That was the single initial snapshot from April 2, and nothing after it.

For the three weeks before the rename, the weekly backup had not been running. The cron had been cd-ing into the hive path (which still resolved pre-rename) but the restic command inside was failing on a credential or path issue, and cron was dutifully writing the error to a log nobody read.

The Chrome debug-protocol scraper that fed our usage metrics had the same story. “Chrome CDP not reachable on port 9222,” every thirty minutes, for weeks. The systemd service that would have started the debug port had never been re-started after a reboot.

Without the migration we would not have discovered either failure. The crons would have kept “running.” The systemd units would have stayed “enabled.” We would have kept believing we had working backups and working metrics.

Infrastructure with no external users complaining decays, and the only reliable signal is an explicit audit.

The dormant-project scanner

On top of cleaning the hive-shaped debris, we wanted something structural so the pattern wouldn’t re-accumulate. We wrote a small Python scanner, scripts/projects/status.py in ops-repo. It walks ~/projects/, reads git metadata per repo, greps ~/.config/systemd/user/*.service and crontab for leftover references to each repo, joins against a hand-maintained intent registry (projects/registry.yaml), and writes a markdown report.

The first run classified 23 repos: 10 active, 2 client, 7 dormant, 4 frozen. The design choice that matters is that unregistered repos surface in their own “needs classification” section rather than being silently omitted. When a new project appears in ~/projects/, the next scan forces a registry decision.

That exact failure mode, the one that produced our previous generations of parked-and-forgotten repos, now has a structural tripwire.

The first scan also caught two repos we had forgotten about, a couple of half-finished side projects, each with leftover user systemd units pointing at its own path. These weren’t hive collateral. It was independent decay: things we started and didn’t finish, whose infrastructure had outlived our attention. The registry entries now record them as dormant with pickup hints.

What this pattern is good for

Two things become possible once the substrate is clean and the boundaries are drawn.

A portable operator surface. The ops-repo pattern is extractable. A sanitized starter version can be handed to a collaborator as a template, letting them open Claude Code in their own operator repo with their own memory accumulating. No shared environment, no shared sprawl. We were making the shape portable to other people.

A place to put durable knowledge that outlives the session. Auto-memory, as Claude Code ships it, is per-cwd and session-accumulated. Next to it, in logs-repo/knowledge/, we’ve started a curated folder of engineering knowledge pages, written like a small wiki, referenced by name from future sessions. It currently holds 3 pages and a README. The policy is to port on demand: when a future session reaches for a pattern that matches an unported learning, we synthesize it then.

What we’d tell the next person doing this

Restructure instead of restarting. A restart tries to solve a sprawl problem with a blank slate and produces abandoned generations. A restructure separates concerns inside the material you already have and produces a portfolio with explicit lifetimes.

Freeze the old thing and keep it. The frozen old engine costs almost nothing to keep. Losing the context, the prior-art and the receipts is permanent. Our old hive is now hive-archive, unchanged, readable, never modified.

Run the audit even if you don’t migrate. The dead backup was the most useful discovery of the whole day, and we would not have gone looking for it on its own merits. If you haven’t traced every cron, every user unit, every background process to its actual current behavior in the last month, you probably have something broken right now. The signal will not come to you. You have to go get it.

Scheduled automation is not evidence of working automation.

dxdev LESSON · paste into your AI coding agent
LESSON: Silent cron and systemd rot after a repo restructure
SOURCE: dxdev.com/blog/backups-dead-three-weeks
WHAT HAPPENED: A developer split a monorepo into sibling repos and updated all internal path references. External automation outside the repo (crontab entries, systemd user services) still referenced the old working directory. The jobs kept firing on schedule, hit path or credential errors, wrote failures to unmonitored logs, and appeared healthy. Backups and metrics scrapers stopped silently for weeks. The failure was discovered by accident during cleanup, not by any alert.
THE RULE: After any repo rename, split, or archive, audit every external reference to that path, not just files inside the repo.
CHECK MY CODE, then report PASS / FAIL with file:line for each:
  1. Any crontab entries (crontab -l) reference a directory or script path that no longer exists or has moved.
  2. Any ~/.config/systemd/user/*.service or /etc/systemd/system/*.service files have a WorkingDirectory= or ExecStart= pointing at a path that no longer exists.
  3. Any scheduled or background job (cron, systemd, Docker, CI) lacks a monitoring or alerting mechanism such that silent failure would go undetected for days or weeks.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.