---
title: "Why our Haiku credits produced 6,134 files that no screen displayed"
canonical: https://dxdev.com/blog/substrate-without-consumer/
datePublished: 2026-05-10
---
Nothing was reading the 6,134 enrichment sidecars we had built for our multi-agent system.

We had spent a sum of Haiku API credits over 11 hours processing our conversation archive. The data was structured, tagged and sitting on disk, and none of it showed up anywhere we looked.

## What the pipeline did

The pipeline took chat sessions from three different AI tools and parsed, classified and enriched them on a nightly cron. The question that stopped it came mid-session, while we were resuming an enrichment run. It wasn't whether the pipeline was well built. It was whether a human would ever see the output. We stopped two enrichment runs mid-flight to find out. A few hundred more sidecars had landed by then, so that work wasn't wasted.

## The missing wire

The sidecars were meant to feed a daily calendar view that aggregates cross-agent activity. The script generating those day-files was still counting raw session manifests. It never read the enriched index. The index existed, and so did the surface we look at every day, but the wire between them did not.

We had built a large producer and skipped the consumer. It's easy to do when you work with agents. You spin up a batch job, push a prompt file, and come back to thousands of processed records in the time it takes to make coffee. That speed feels like progress. Pointed at the wrong target, it is just faster waste.

## The fix was 40 lines of Python

When we audited the system, the fix was almost embarrassingly small. The day-file generator already had a placeholder block for cross-agent data. The missing piece was about 40 lines of Python to load the index, group the entries by provider and render a simple table. That is a zero-cost, one-afternoon task.

Because we hadn't built it, the whole enrichment run was dead weight. We stopped the runs as soon as we saw that nothing consumed the output. There is no point in processing the rest of the corpus until the wire is in place. The limit on this project was never data or API quota. It was our own attention, and the surfaces we use to make decisions.

## Sequencing the work

Building a producer feels like progress. You are processing data, generating output and filling directories with structured files. It looks like work. But if nothing reads those files, you have built a very expensive buffer.

So we are changing the order. The consumer surface gets built first, as a working view fed by mock data if it has to be. Get the view working and confirm the data it would show is useful. Only then build the pipeline that populates it. That way you prove the value before you pay for the infrastructure. If you can't build the consumer yet, don't build the producer.

The same check applies to any data store, processing layer or enrichment system: what is the first surface that will read this? If you can't name it, you aren't ready to build the producer.

The index is clean, the day-file generator has its placeholder, and the sidecars are waiting. The 40-line wire is the only thing between that pile of dead weight and a working feature, and it is the next thing we build.