1,586 Claude sidecars, and the calendar page mattered more than the search index

The first pass over 1,586 Claude conversation sidecars wrote only 847 rows and printed no error, because a helper returned None where the database column required a string and 715 row writes rolled back without a word. I noticed only because the count was short. That silent gap nearly made the search index I was building untrustworthy, and it sat in front of a bigger question: what did I work on three weeks ago, when the only way to answer was to open folders by year, month, and session identifier and read filenames? A nightly calendar page ended up answering that question better than the index did.

The enricher came before the index

Claude writes a sidecar for each session. Each file contains conversation turns, metadata, timestamps, and the context present at the start of the session. The 1,586 sidecars spanned roughly one year.

My first instinct was a search index. An index assumes data that is consistent enough to query, and the sidecars were not. Older session hooks had used different shapes. An area classification could be absent, stored under an earlier field name, or stored under its later field name. The sidecar writer also failed to sanitize Claude’s output before serializing it, leaving malformed JSON in summary values. Indexing those files without normalization could return a partial answer without showing which sessions had been omitted.

The enricher had a narrower job. It extracted the timestamp, resolved the area across those source shapes, pulled themes and ticket references, and wrote one structured row for each usable session into a database with a controlled schema.

The first pass wrote 847 rows with no visible error output. It silently failed on 715 sidecars because a helper returned None instead of an empty string when a session lacked an area and the destination column required a value. The row write rolled back, the error did not appear in the normal output, and I caught the failure only because the row count was wrong. The fix made a short write visible: each skipped record receives a reason, and the final row count is compared with the number of files processed.

The aggregator surfaced the work I minimized

Once the enricher was stable, I built a theme aggregator that counts recurring topics across a date range. Each session creates a theme list during enrichment. A single list is noisy. A weekly aggregate shows which topics actually consumed the sessions.

When I ran it over May, browser automation on the legacy platform accounted for 10 percent of the sessions. The slice contained eight sessions. I had treated that work as cleanup between larger tasks. The aggregator put a number on work I was discounting, and the eight-session slice made the pattern harder to dismiss.

Calendar pages mattered more than search

The calendar-day composer began as a secondary feature after search. It became the primary one.

The composer runs at night, gathers enriched sessions for one calendar day, and writes a narrative Markdown page with the work, themes, and daily arc. It creates one file each day at archive/<year>/<month>/<day>/SUMMARY.md.

Search answers a narrow retrieval question. A calendar page answers what was happening on a particular day, which is the question I usually need when reconstructing context for a follow-up or writing a ticket-tracker comment. The page has a through-line rather than a ranked result list.

The first composer drafts were accurate and flat. They recorded session totals and ticket counts but did not show what moved during the day. I changed the prompt to ask for the day’s main arc, where it started, where it ended, and what shifted. The second drafts were better.

The nightly schedule keeps the archive current

The nightly schedule runs at 2 AM through the host scheduler. It runs a refresh script that survives reboots and records its output in logs/nightly-refresh.log.

Before the schedule, the calendar pages came from one manual backfill. After the schedule, the archive gains a page every night. A memory system that requires a remembered manual command will not remain current.

The archive sidebar now links to every day. Days with summaries show the theme cloud, and days without summaries show the raw session count. I can scan a month of work in 30 seconds.

The missing pieces are still visible

The ticket-tracker connection is incomplete. The enricher extracts ticket references from session context but does not write them back to the ticket tracker. The intended loop is small: enrich a session, identify the ticket it touched, and add a short reference to the session history.

Cross-agent counts are incomplete too. The enricher counts Claude sessions but does not count Manus sessions or Codex runs because those runs do not write sidecars. Until those counts exist, I cannot size that work properly.

The first complete pass reached 1,562 of 1,586 sidecars. The remaining 24 had corrupt timestamps, and the list is known rather than queued.

The 24 timestamp-corrupt sidecars remain in the archive, unenriched, while the nightly schedule keeps writing calendar summaries.

LESSON · paste into your AI coding agent
LESSON: Silent partial success in a batch enricher corrupts every query downstream
SOURCE: this post
WHAT HAPPENED: A batch enricher processed structured conversation records and wrote only part of the expected corpus without visible error. A helper returned None for a required field, causing row writes to roll back. The problem became visible only when the output row count was lower than the processed-file count. Queries built on the resulting table used an incomplete corpus.
THE RULE: A batch enricher must expose partial success by logging each skipped record, comparing rows written with files processed, and failing when the counts do not reconcile.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any database write that can fail without recording the record identifier and reason.
  2. Any batch process without a final comparison between files processed and rows written.
  3. Any required destination field without an explicit fallback before the insert.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.