---
title: "Five Hours Chasing 2000.0 Minutes While One Annotation Sat Unread"
canonical: https://dxdev.com/blog/ci-annotation-not-log/
datePublished: 2026-08-29
---
The alert said the failure was a cache issue. The job had run zero steps, in two seconds, twice in a row, and the actual reason was sitting in a GitHub check-run annotation our triage script never read.

## An alert that guessed

`ci-alert/triage` fires on red CI. That morning it fired three times between 9:47 and 1:26, and each time it produced a plausible-sounding cause. All three were wrong in the same way. They were guesses generated from an error blob, and GitHub hadn't told us any of them.

Here is what `ci_alert.py` was doing. On a failed run, it called `gh run view --log-failed` to pull the failing step's log and fed that text to a triage model. That works when a step actually ran and printed something. It breaks when the job never started, because there is no failing step and no log. In this instance the command returned an Azure `BlobNotFound` document, since the log artifact for a job that never ran doesn't exist. The script had no check for that. It handed the model a storage-service error page and asked it to name a cause, and the model invented one. That is how we got a believable cache-related root cause for a job that had nothing to do with caching.

## Pulling the real data by hand

We went and pulled the real data instead of trusting the alert:

```
gh api repos/OWNER/REPO/commits/SHA/check-runs \
  --jq '.check_runs[] | {name,conclusion,output_title,output_summary}'
```

That gave a check run with `conclusion: failure` but an empty `output_summary`, so we went one level deeper, to the annotations on that check run:

```
gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID/annotations
```

There it was, `annotation_level: failure`, with the message GitHub actually wrote:

> "The job was not started because recent account payments have failed or your spending limit needs to be increased. Please check the 'Billing & plans' section in your settings"

Re-running it as attempt 2 produced the identical annotation. It wasn't transient and it wasn't a code problem. GitHub Actions had been switched off at the account level because a usage allowance had been hit.

## The detour through the billing API

Our first instinct was that this had to be a billing question, so we went for `gh api users/OWNER/settings/billing/actions`. That came back `410 This endpoint has been moved`. We tried the replacement, `users/OWNER/settings/billing/usage`, and got a different failure: `invalid API endpoint: "C:/Program Files/Git/users/OWNER/settings/billing/usage"`. Git Bash on Windows was rewriting the leading slash into a filesystem path. We stripped the slash, got the endpoint working, and pulled back a usage report. It confirmed the account had hit exactly 2000.0 minutes for the month, all of it from one repo's deploy-on-every-push workflow.

That number was real and useful for the follow-up conversation about trimming that workflow. It told us nothing about why this specific job had failed. We spent the better part of the diagnosis on account-level usage data while the cause sat in the check-run annotation for the commit in question, one API call away, in GitHub's own words. The detour is the reason this ticket took over five hours instead of the twenty minutes the annotation lookup took once we ran it.

## Routing the zero-step case to annotations

The triage function already had a comment naming this exact shape of bug: `startup_failure with no failing step` was called out as a known case. The code path that would handle it never called the check-run annotation lookup. The fix was to make the zero-step case go there before falling back to logs:

```python
text = _gh_cli(["run", "view", str(run_id), "--repo", owner_repo, "--log-failed"])
if text.strip():
    return _error_window(text, cap)
jobs = _failing_jobs(owner_repo, run_id)
if _never_started(jobs):
    ann = _job_annotations(owner_repo, jobs, cap)
    if ann:
        return ann
return _whole_job_logs(owner_repo, jobs, cap)
```

`_never_started` checks for the zero-steps signature and, when true, pulls annotations before ever touching a log. We also tightened the triage prompt. If the evidence is a GitHub annotation saying the job ran no steps, the model is now told to report that reason in GitHub's own words as authoritative, and to stop inferring a cause from repo lines that are just environment noise (DB roles, connection strings, checkout, teardown).

Cost stayed at one `gh` call in the common case. Both extra lookups sit past the `--log-failed` early return, so a run with a normal failing step still makes exactly one call.

We verified it end to end against a real red run: same annotation, same message, now surfaced directly instead of paraphrased from a blob. The root cause was reported correctly. GitHub had halted Actions on the account over payments, and the code was fine. We moved the offending repo to a separate billing account and left the rest waiting on the month to flip.

## What to check in your own alerting

When a CI job fails with no steps recorded and nothing useful in its log, ask for the check-run annotation before accepting any root cause from an alert tool. GitHub writes down why a job did not start, in its own words, in a different API from the logs. Check that your alerting reads that source, and treat a cause generated from a storage-service error page as a guess.