The 3:41 Site Alerts That Pointed at the Wrong Layer

Four minutes was enough to turn a read-only agent session into a production incident. An autonomous session ran an unbounded scan against our shared production database, and the live site stalled before the session had an answer.

At 3:41 that afternoon we started from the site-error alerts. The symptom the alerts showed was generic application failure. The cause sat lower in the stack: one open-ended database read had wedged the database. The session had asked a question with no hard boundary around the work needed to answer it.

A query can leave every row intact and still eat enough locking or input/output capacity to take a live service down with it. We had made production data available because agents need real state to investigate real problems, and I think that is still the right call. But we had treated database access as a permission question. The incident showed it is also an execution-budget question.

Within 91 minutes we had shipped and live-verified three controls: a machine-level block on unbounded scans, a bounded query wrapper with timeouts, and a monitoring change that treats database stalls as an alertable operational failure.

What the alert told us and what it was

Our monitoring described the symptom too loosely. Error alerts from the live site said requests were failing. They did not say why.

Tracing led to the shared production database, and from there to the unbounded scan from the other autonomous session. The chain was short once we were looking at the right layer:

What we observed	What it first told us	What it was
Site-error alerts	Requests were failing	The database was wedged under an unbounded scan
No rollback target	It was not obviously a release problem	The failure came from a live query path
A read-only session	No data should have been changing	A read can still saturate the resources serving live requests

This is why our access rules for autonomous sessions can’t stop at read versus write. That split protects data integrity. It does nothing for availability.

Moving the rule into the access path

A stronger prompt would leave the same failure depending on an agent remembering a policy under every new task. Manual query review would remove the autonomous part of an autonomous session. Monitoring alone would only tell us the site was already in trouble. So the control had to run before the database did.

The first layer is a machine-level block on unbounded scans. What matters is where it lives. The agent can’t decide mid-investigation that its own question deserves an exception. If a request has the shape of an unbounded scan, it is stopped before it reaches the shared production database.

That gate also caught our own build-time test payload, which is the kind of verification we like, because it shows the block really does intercept the dangerous path.

The second layer is the bounded query wrapper. Production reads go through a route with explicit limits and timeouts instead of an open-ended connection. A query that can’t finish within its budget doesn’t get to keep spending the live service’s budget.

The two overlap on purpose. The block handles the class of request we now know is unacceptable. The wrapper limits the work of requests that are otherwise allowed.

Database stalls are now incidents

The third layer is monitoring. We upgraded the site watcher to sample blocked and long-running SQL every minute. Lock and input/output incidents now produce capture and alerting, where before we would have inferred them later from generic application errors.

It isn’t glamorous, but it closes the loop. CPU spikes already get operational attention because they are visible and legible. A database wedge needs the same treatment. If the database is blocked, or a query runs long enough to threaten the site, that is an incident signal on its own.

The response took 91 minutes from the first trace to shipped, live-verified guardrails. The follow-on port for other operators was queued as separate work. We didn’t call the job done because one machine had a new rule. The useful unit is the access pattern, not the one host where we first learned the lesson.

We still let agents query production data. We don’t let an agent turn an investigative question into an unlimited claim on the database that serves live traffic. Those are different capabilities.

What to check

If agents can read your production database, check three things. Do production reads go through enforced limits and timeouts? Is an unbounded query blocked before it reaches the shared database? Does a blocked or long-running query raise an alert as an incident of its own?

dxdev LESSON · paste into your AI coding agent
LESSON: Read access is not harmless access; an unbounded query can wedge a live database without changing a single row
SOURCE: dxdev.com/blog/agents-on-prod-require-active-guards
WHAT HAPPENED: An autonomous agent session, granted read access to investigate a real problem, ran an open-ended database scan with no bound on the work required to answer its question, and the resulting lock and I/O load stalled the live site within four minutes, even though no data was changed. The site-error alerts that first appeared described only the symptom, application failures, not the actual cause sitting one layer lower, a wedged database. Within 91 minutes, three controls shipped and were live-verified: a machine-level block that stops any request shaped like an unbounded scan before it reaches the shared database, a bounded query wrapper that gives production reads explicit limits and timeouts instead of an open-ended connection, and a monitoring change that samples blocked and long-running queries every minute so a database stall is now its own alertable incident rather than something inferred later from generic errors.
THE RULE: Read versus write access protects data integrity; it does not protect availability. A query can leave every row untouched and still consume enough locking or I/O capacity to take down the service it's supposed to be investigating, so an execution-budget control (bounded queries, blocked-request detection) has to run before the database sees the request, not depend on the agent remembering a policy in the moment.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Any autonomous or scripted process granted read access to a shared production resource with no hard bound (timeout, row limit, cost limit) on the work a single query can request.
      2. Any access-control model that stops at read versus write, without a separate control for the execution cost or resource consumption a read can impose on a live service.
      3. Any monitoring setup that infers a database or resource stall only from downstream symptoms (generic application errors) rather than sampling the resource's own health directly as its own incident signal.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.