At 12:06 AM, the first investigation our nightly loop generated closed, and it was a scanner false positive.
Our agent systems were producing more edge cases than we could hunt by hand. The evidence was scattered across transcripts, investigation files, task history, and commits. One isolated failure might be a prompt oddity. The same failure pattern appearing across runs is a candidate for a durable fix.
So we built a loop around that distinction. Every night, a scheduled job mines the record of agent work for lessons. When it finds a recurring failure pattern, it files an investigation instead of leaving a loose observation in a transcript. The investigation can query indexed git history, which lets it ask something a transcript alone cannot answer: is this behavior still present, or did a later commit already fix it?
A false positive on night one was the result we wanted. We did not want a bad signal, but false positives are where an automated feedback system proves whether it is useful. A system that only describes its own successes is a reporting layer. A system that can catch a bad read, open the work, correct the scanner, verify the correction, and close the investigation has started to become operational.
The results flow into a self-improvement console. The console is the evidence trail, and the product is the loop from recurring pattern to investigation, diagnosis, change, verification, and closure.
Why a summary or a dashboard would not have closed anything
The sequence is intentionally simple:
nightly schedule -> mine agent transcripts for recurring failure patterns -> file an investigation -> query indexed git history during diagnosis -> make and verify a correction -> record the lesson in the self-improvement consoleThere were easier versions. We could have generated a daily summary of odd agent behavior. We could have put every suspicious event into a dashboard. We could have treated the transcripts as an inbox for manual review.
All three lose at the same point. They preserve information, but they do not create a unit of work with a path to resolution. A summary is easy to skim and easy to forget. A dashboard lets a problem wait in public. Manual pattern hunting means the system only improves when somebody remembers to look for the same shape of failure across separate runs.
Letting the agents auto-repair every anomaly they found would have been worse, and the first night showed why. A finding is not a bug merely because a scanner can name it. The loop needs an investigation stage between detection and change, and verification after the change. That keeps a noisy detector from turning into a noisy repair bot.
We kept the git index inside the loop for the same reason. Transcripts can show that agents repeatedly hit a condition. They cannot establish whether the repository now contains the answer. Searchable history lets the investigation separate an unresolved pattern from one already superseded. Without that step, the system would keep reopening dead work and calling it improvement.
What the first finding turned out to be
The initial investigation did not uncover a new product defect. It uncovered a bad scanner signal.
The scanner had produced something that looked like a recurring failure. The investigation found that every flagged hit was a synthetic plumbing-test prompt, one of our own automated test seeds, and not a real correction the system had needed to make. We added a content blocklist to the scanner. A re-scan confirmed the fix: the flagged pattern went from three hits in the prior week to zero.
The build took five minutes, starting at 12:01. The work log also records a separate follow-up filed the same night on flagged spend paths, unrelated to the false positive. That matters because the loop did not turn one bad signal into a reason to distrust every signal. It separated the incorrect read from other work that still needed a tracked follow-up.
A feedback loop should not be so fragile that one false alarm poisons it. It should be able to say, with a record behind it, that this finding was wrong, this component was wrong, here is the fix, and here is the verification.
Counting closed loops
The tempting metric for agent systems is output volume. How many transcripts were mined, how many lessons extracted, how many tickets created. Those are activity metrics, and they do not tell us whether the system got better.
The useful unit is the closed loop. A recurring pattern becomes an investigation. The investigation is checked against the code and history. A change is made only when the diagnosis supports it. The result is verified, and the record makes the next diagnosis better. In this case, the first completed cycle improved the scanner that feeds the loop.
That is a narrow result, but it is real. We now have a nightly process that turns accumulated agent behavior into investigated work, uses repository history as part of the diagnosis, and leaves behind a verifiable correction instead of a theory about what the agents might be doing wrong.
The re-scan is the whole test of whether a loop like this is real. Could it prove, with a number, that the problem was gone? Three hits down to zero did that. If you automate detection of problems in agent work, put an investigation step and a verification step between the finding and the fix, and ask for a before and after number on every correction. A detector that cannot be corrected and re-checked will produce noise, and acting on that noise is the risk.