A test_units_store p95 check with a 20 ms budget passed on the best of up to five rounds, so an intermittent slowdown passed it too.
Best-of-5 went in with the other flake fixes
The change to the units_store p95 test was one item in a commit about nightly flakes, which the log counts as “3 test bugs and 1 real gap”. The other items:
test_compact_stages and test_refresh_lock_masking computed time at import, so they now compute it per test.
test_signin_hold got a wait of 120s.
commit_scope_guard got a git timeout of 600s.
The units_store p95 test moved to best-of-5 rounds, with the 20 ms budget unchanged.
The real gap was in hook_block_pretransition_edit. Its git probe timed out at 3s and fell open under load. It now waits 15s and fails closed, with 3 new tests.
The review at 4:55 pm
A review of that commit came back at 4:55 pm with a medium finding on line 59 of the test file. The finding was about taking the minimum p95 across up to five rounds. That turns a p95 performance budget into a different check, because one fast round is enough to pass.
The median of five fixed rounds
A separate commit asserts on the median of 5 fixed rounds instead of best-of-5, with the 20 ms budget unchanged. The old assertion took the minimum p95 across up to five rounds. The new one always runs five and holds the middle one to 20 ms, so an intermittent slowdown no longer passes.
AI Skills
Use this lesson with the AI assistant you already use
A test_units_store p95 check with a 20 ms budget was changed to pass on the best of up to five rounds, so an intermittent slowdown could pass it. A later commit replaced that with the median of five fixed rounds, keeping the 20 ms budget.
Paste the prompt, share only the context needed to answer it, and treat the result as a draft for your review. Do not include confidential information or let an AI assistant make changes without your approval.
The prompt works in any AI assistant as it is. We are also building a /dxdev skill for visual reports and saved memory.
It is not released yet: see what you can use today.
dxdev LESSON · paste into your AI coding agent
LESSON: A Performance Budget Must Judge Every Run Or An Intermittent Slowdown Will Pass It
SOURCE: dxdev.com/blog/best-of-five-hid-a-slowdown
WHAT HAPPENED: A commit about nightly flakes, counted in the log as 3 test bugs and 1 real gap, moved the units_store p95 test to best-of-5 rounds while leaving the 20 ms budget unchanged. A review of that commit came back at 4:55 pm with a medium finding on line 59 of the test file. It flagged that taking the minimum p95 across up to five rounds changes what the check measures, because one fast round is enough to pass. A separate commit then asserted on the median of 5 fixed rounds instead. That version always runs five rounds and holds the middle one to 20 ms, so an intermittent slowdown no longer passes.
THE RULE: When you retry a performance test to cut flakiness, do not assert on the best result, because that hides real intermittent slowdowns. Run a fixed number of rounds and assert on a statistic, such as the median, that a recurring slowdown can still move.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
1. Does any timing assertion take the minimum or best of several retries, or stop at the first passing round, so that one fast round can hide slow ones?
2. Does the test run a fixed number of rounds every time and assert on a statistic like the median, rather than a variable number of rounds?
3. When a flake fix touches a performance test, is the budget left unchanged while the assertion is checked to confirm it can still fail on an intermittent slowdown?
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.
