Marking Up a GPT Plan at 10:47 PM Until Dawn

At 10:47 PM we sat down with a GPT-authored strategy document open on one screen and the live archive open on the other, and started marking the document wrong, one recommendation at a time. Six hours and eleven minutes later we had rewritten most of it, written down a canonical ticket workflow that had never existed on paper before that night, and found two real bugs in how our agent sessions put finished work in front of us to review. None of those three things were the plan when we opened the laptop.

Where the strategy doc went wrong

Earlier that night, at 1:42 AM, a different agent had already reviewed the same document and returned a conditional verdict. Approve the measurement direction, but do not proceed dashboard-first or checklist-first. That is a sequencing call, not a rejection of the whole plan. The document’s instinct to build something visible to show progress was right. Its order of operations was backwards.

The GPT plan wanted a dashboard first, something you could point at and show someone. What needed to happen first was checkout-date tracking and a reliable snapshot underneath it. We had already shipped the checklist and checkout-date hotfixes days earlier, and the nightly metrics snapshot had only just gone live that night. Build a dashboard on top of that and every number on it means something. Build it first and you have shipped a page that visualizes nothing.

So we went through the document against the archive and our own history of past ticket decisions, correcting it recommendation by recommendation. We also wrote down for the first time our stance on agent-system investment: fund the plumbing before the display. That is in the archive now, for the next session that has this same argument with itself.

The blank page

With the sequencing settled, we built the front end: a Conversion page and release markers, on top of the snapshot data that was now real. The standard move to close the loop is to point the agent browser at the new page, capture it, and have the capture waiting for us.

It came back blank. A page shell, no chart, no rows, exactly what you would see if the metrics endpoint had never been called. We re-ran it. Blank again. We assumed it was a data problem and checked the nightly snapshot job directly against its output in the database. The data was there.

The fault sat in the browser session. Earlier that same night we had fixed the agent browser to sign in to platform staff automatically, so sessions would not need to be walked through login by hand every time. That auto sign-in was writing its session cookie against a fixed QA tenant regardless of which environment the target page lived on. The Conversion page was rendering under a test account that had never run the checklist flow, so the query it fired came back empty, correctly. The login shortcut that was supposed to save time was authenticating every capture against the wrong tenant.

We fixed the host binding and reran the capture. A second bug showed up immediately. The screenshot fired before the page’s async metrics fetch had resolved, so now we had a page correctly authenticated against real data, captured mid-spinner. That one was a straight race. The browser tool took its shot on page-load instead of network-idle, and a chart that fetches after mount loses that race close to every time.

Neither bug had anything to do with conversion metrics logic. Both live in the layer that takes what a session built and puts it in front of a human to look at, and neither had ever been caught by a unit test, because nothing about them is a unit. They only show up when a real page, with a real chart, gets rendered through the actual login path a real session uses. We filed both as follow-up tickets instead of patching them inline, because the auto sign-in host binding affects every other page any agent might screenshot for review, and this page is only one of them.

What we kept

The GPT document was worth writing. It got a plausible plan down fast and gave a second agent and us something concrete to disagree with, which is faster than starting from nothing. The value came after that. Correcting it against real history produced a written policy that had not existed before that night, and the unrelated task of getting a finished page onto a screen surfaced two bugs that had nothing to do with conversion metrics at all. We went looking for a dashboard and came back with a rule about session hosts and a rule about screenshot timing. Both are more durable than the dashboard was.

If you’re in the same spot

When an AI-written plan proposes a dashboard first, ask what data sits underneath it and whether that data is reliable yet. Review any such plan against your own decision history before acting on it. If automated page captures come back blank, check which account and environment the capture session signed in under before you suspect the data.

dxdev LESSON · paste into your AI coding agent
LESSON: Build Measurement Plumbing Before Its Display Layer
SOURCE: dxdev.com/blog/ai-collaboration-exposes-tools
WHAT HAPPENED: A second agent approved the document's measurement direction but rejected its dashboard-first sequencing. The author compared the plan with the vault and prior ticket decisions, then documented a canonical workflow and the policy to fund underlying plumbing before visible reporting. Once the Conversion page was built on real snapshot data, its first browser captures were blank because automatic staff sign-in always used a fixed QA tenant. After correcting the host binding, captures still occurred during the loading spinner because screenshots ran on page load before the asynchronous metrics request finished. Both defects were filed as follow-up tickets because they affected the shared review path for any agent-rendered page.
THE RULE: Make the data collection and reliable snapshot real before building the dashboard that presents them. Audit the end-to-end review path with real authentication and asynchronous page behavior, because shared session tooling can invalidate otherwise correct work.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Every dashboard metric is backed by a populated, scheduled or otherwise reliable source snapshot before the dashboard is treated as complete.
      2. Automatic browser sign-in binds its session to the host or tenant of the page being captured rather than a fixed environment.
      3. Screenshot capture waits for the page's relevant asynchronous data request or a defined ready state, not merely initial page load.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.