What the Tracking Screen Showed for 205 Customer-Managed Domains

The morning the second reminder email went to all 205 customer-managed domains that still had not moved over, we found that the check behind our tracking screen had never looked at about a third of them.

A domain is the web address a customer’s site lives at. Those 205 were ones our customers manage themselves that still pointed at the old setup. A nightly check visits each one and records whether it has moved. The screen shows the results. We had been reading that screen as the whole list, because nothing on it said otherwise.

The check that stopped without saying so

The nightly check has a one-hour limit. It was hitting that limit and stopping, with no error, no warning and no record that it had quit early. Every site it reached had a row on the screen, and those rows looked like proof of coverage. They only showed that some work got done before the clock ran out.

Shipping the reworded warnings, sending the email and finding this all fit inside one 39 minute session that ended at 8:56. We filed the finding along with three other follow-ups and closed it out.

One blank, three meanings

A site with no problem showing could mean it was checked and fine. It could mean it was checked and still needs to move. Or it could mean the check never got to it. The screen could separate the first two. It could not separate the third from either of them.

For a campaign whose whole point is who still needs to move, that third case is the entire question.

Why a longer limit was the wrong fix

The obvious fix is a longer time limit. That only moves the line. A bigger limit still cuts the check off before the list is covered, and now it happens on a busier night when nobody is looking.

What the check has to record is three things: how many sites it expected to cover, how many it reached, and whether it finished. Only when all three agree does the list count as coverage. That day’s commit log also shows a change so the scan raises an alert when it gets killed. That silence is what started all this.

What Claude can and can’t catch here

Claude is quick at reading stored results and counting them against a list. That is the question that exposes this kind of gap: whether every one of the 205 is accounted for by a run that finished. It only answers that if someone asks it to count expected against actual. Asked to summarise what the screen showed, it would have summarised a third of a list as if it were the whole list.

Who receives a message is a decision about people, so a person has to decide whether the list behind it is complete. An assistant can draft the email and tidy the list. It cannot know the list was cut off unless the check says so.

What a check owes you before you send

A check that a clock can stop has to say how far it got: expected 205, reached some number, finished or not. Until it says that, the rows it left behind prove that work happened and nothing more. Before sending anything to a list, find out whether the check behind it records how many it expected, how many it reached and whether it finished.

dxdev LESSON · paste into your AI agent

LESSON: A blank result can mean never checked, and a timeout makes that invisible

WHAT TO SHARE:
A de-identified description of a scheduled check, audit or report that covers a list of things, how long it is allowed to run, and what a person decides from its output.

ASK YOUR AGENT TO:
1. Say what happens to the check if it hits its time or resource limit before finishing. Does anything record that it stopped early?
2. For an empty or missing result in the report, list every reason it could be empty, including the item never having been reached.
3. Compare the number of items the check was meant to cover with the number that have a result, and name the items with no result.
4. Flag any decision, such as who gets a message, that is made from the report without first confirming the check finished.

RETURN:
A short list of the ways this check can look complete without being complete, and the three numbers that would prove it: expected, reached, and whether it finished.

BOUNDARY: Do not change the schedule, the time limit or the report. A person reviews the list and decides what the check should record.