The Run That Finally Exited 4 After Every Run Exited 0

The run that finally exited 4

Exit code 4 came back on the first run where the meter was actually allowed to say no. Before that, every run of the tracking harness had exited 0, and every 0 had a number attached: translation error in pixels, lag in milliseconds, comfortably inside the budget we’d written down before we ran anything. The 0 was wrong. It took an adversarial read of the meter itself to prove it.

The harness is three Win32 processes on one virtual desktop. Target is a plain window that gets moved around, by script or by hand. Tracker is the layered overlay under test, a strip that’s supposed to sit flush above the target no matter where it goes. Meter is the judge, and it does not call GetWindowRect on anything. It uses DXGI Desktop Duplication to capture the composited desktop, hunts for four 16x16 fiducial markers, two on the target, two on the overlay, and computes translation error, edge error and size error from where the markers actually landed on the glass. The whole point was to stop trusting Win32 coordinates and start trusting pixels.

The fix that only fixed what we already knew

The meter’s acquire loop originally did three things on one thread: call AcquireNextFrame, map the staging texture, and once a second, sample the tracker’s and target’s process stats for the CPU/GC/memory budget. The first matrix run came back clean, and also came back with a pattern we almost let slide: every frame the log marked coalesced landed on an integer-second boundary. On it, every time.

A blocked capture thread produces exactly that pattern. The once-a-second stat sample was long enough to make AcquireNextFrame miss the next composited frame, and DXGI’s response to a slow consumer is silence: no error, no missed-frame count, nothing in the return value that flags it. It just hands you the next available frame with whatever happened in between folded in. We split the meter into three threads: acquire-only, a worker that maps (D3D11_MAP_FLAG_DO_NOT_WAIT, retried) and detects, and a separate 1 Hz sampler, so nothing but frame acquisition ever touches the acquire thread. That killed the boundary artifact. We treated it as the fix.

It fixed the case where we already knew to look for it.

Why the missing frames were the ones that mattered

Desktop Duplication doesn’t queue frames. AcquireNextFrame gives you the current composited state, and if you were too slow to ask for the previous one, it folds the missed updates into whatever it hands you next. There’s no counter for that. The only trace is that the frame you got spans a longer interval than the nominal refresh, and unless you’re checking against something outside the meter’s own clock, you don’t know it happened.

A review pass on the meter’s methodology put the failure mode in front of us: a bias that points in exactly the wrong direction. Coverage is worst precisely when the target is moving fastest, because a moving target generates composited updates at a higher rate, which leaves the worker thread less time per frame to map and detect before the next one lands. Detection under motion is also more expensive, since more of the region is plausibly a marker mid-transit and the tolerance search does more comparisons. So the frames most likely to get silently coalesced away are the ones from the segments where the tracker is most likely to be lagging, and the frames that survive to get measured skew toward the calmer stretches where the overlay already had time to settle. Average the pixel error over whatever frames.csv happens to contain and you get a number that looks great, right up until you watch the overlay on screen and see it stutter through a resize.

We had built an instrument that would report a near-perfect score for exactly the failure it existed to catch.

Making coverage a pass condition

A faster meter would not have fixed it. Detection cost scales with marker count and search tolerance, and we weren’t going to win a race against the compositor. The fix was making the instrument refuse to answer when it couldn’t see.

Meter now takes --target-log and --tracker-log, the same per-move logs Target and Tracker already write for their own diagnostics: kind 1 for a commanded move, kind 2 for WM_WINDOWPOSCHANGED, and so on down the list. Every gap in the captured-frame sequence gets checked against both. The target log gives the commanded velocity and amplitude during the gap, so the meter can compute the worst-case position error that gap could be hiding, and the rule is that every commanded move has to show up in a captured frame within 2.5 composited intervals or the gap gets charged against that bound. Without the tracker log, the meter doesn’t get to assume the overlay held still either. It assumes the overlay was frozen for the whole gap, the worst case, the inconvenient one.

That bound is coverage’s teeth. No capture percentage gets eyeballed and called close enough. For every stretch the meter didn’t see, it asks what the largest error that gap could be hiding is, and whether that worst case still clears tolerance. If it doesn’t, the run doesn’t get to report a pixel number at all. Meter exits 0 when the lag and jitter claim is actually supported by coverage, 4 when it isn’t, 2 on setup failure. Exit 4 means the meter declined to answer.

The first run under the new rule came back exit 4. The tracker hadn’t been touched. The accounting of what the meter couldn’t see, bounded against what the target log said was happening during those gaps, exceeded the budget we’d written down before any of this ran. The 0 we’d been looking at for two matrix runs had never been a real 0. It was an average over a sample that had already had its worst evidence removed before we ever saw it.

A measurement that exits clean is only as good as its proof that it saw everything. In this harness, making coverage a pass condition turned a false 0 into an exit 4.

dxdev LESSON · paste into your AI coding agent
LESSON: Treat Unobserved Intervals as Worst Case Evidence Before Claiming a Pass
SOURCE: dxdev.com/blog/coverage-is-a-pass-condition
WHAT HAPPENED: The meter originally acquired frames, mapped textures, detected markers, and sampled process statistics on one thread. Its once per second statistics work blocked capture, producing coalesced frames at integer second boundaries, so the meter was split into acquisition, detection, and sampling threads. A methodology review found that Desktop Duplication silently folds missed updates into later frames, especially during fast motion when detection costs more. This removed the frames most likely to show tracker lag from the average, making the results look better than the visible behavior. The revised meter compares every capture gap with target and tracker logs, bounds the position error that could be hidden, and returns exit 4 instead of a score when coverage cannot support the claim.
THE RULE: Do not accept an accuracy or latency result merely because the observed samples pass. For every interval an instrument fails to observe, bound the worst error that could be hidden there and fail the result when that bound exceeds tolerance.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
      1. Every gap in the captured frame sequence is identified and evaluated against an external event log, not only the meter's own timestamps.
      2. For each capture gap, the process computes a worst case error bound from the commanded movement during that interval.
      3. The test returns no passing accuracy or latency score when any gap's worst case bound exceeds the configured tolerance.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.