The old ListSessions call capped at 200, and 476 agent tasks were missing from our conversation archive.
What the scraper did
The scraper pulls the agent service’s sessions into raw JSON, and a parser turns them into per-thread markdown. The documented refresh step was the scraper with --files --skip-existing, followed by the parser. Discovery went through the internal ListSessions endpoint with a JWT.
A row in our API doc covered what to do about the scraper. It said the scraper stays on the internal JWT for now, because the archive parser depends on internal response shapes. The port to task.list and task.listMessages was to happen “when the JWT next breaks.”
That trigger was a broken JWT. The 200 cap was what had dropped the 476 tasks.
What replaced it
Discovery is now the official task.list. It is paginated, uses the API key, and is free. Bodies still ride the internal JWT getSessionV2. Porting them to task.listMessages would retire the JWT refresh step.
Alongside the discovery change, the scraper picked up these behaviors:
Incremental fetch. It refetches a task when its updated_at changes. The command is now scrape_manus.py --files with no --skip-existing, and it fetches new and changed tasks only.
Shape check and quarantine. Responses are shape-checked before anything is written, and bad ones are quarantined.
Auth failures. A 401 or 403 halts the run with one nudge. The exit code is 2.
Upstream failures. A 5xx is skipped or aborts the run. Exit 3 means upstream is down.
Bounded run and status file. The run is bounded, and counts land in a status file.
Nightly push. The nightly job pushes the single-writer archive repo.
The change touched 10 files, with 1193 insertions and 170 deletions. The scraper alone accounts for 761 changed lines. The new hardening test file adds 367 lines, and the refresh-history test file gains 48.
The webhook item
The same doc had a deferred item for webhooks: a task_stopped event feeding our secretary inbox. It needed a public HTTPS endpoint first. We closed it as superseded. The nightly task.list poll keeps the archive current for free and needs no public endpoint.
The doc row for the scraper used to say it stays on the JWT until the JWT breaks. It now says discovery is task.list, paginated, because the internal ListSessions capped at 200 and had dropped 476 tasks.
AI Skills
Use this lesson with the AI assistant you already use
The scraper used an internal ListSessions call that capped at 200 results, and 476 agent tasks were missing from the conversation archive. The API doc said to port to the official endpoints only when the JWT next broke, so the real problem sat unnoticed.
Paste the prompt, share only the context needed to answer it, and treat the result as a draft for your review. Do not include confidential information or let an AI assistant make changes without your approval.
Optional: for a visual report and saved memory, run /dxdev first.
Don’t have it? Get it at dxdev.com/skills/dxdev. The prompt works without it.
dxdev LESSON · paste into your AI coding agent
LESSON: Replace A Deferred Fix Trigger Of Failure With A Check On Silent Truncation From API Limits
SOURCE: dxdev.com/blog/api-limit-hides-scale
WHAT HAPPENED: The scraper discovered sessions through the internal ListSessions endpoint using a JWT, and a parser turned the raw JSON into per-thread markdown. The API doc deferred porting to task.list and task.listMessages until the JWT next broke. The JWT never broke, but the 200 cap silently dropped 476 tasks from the archive. Discovery now uses the paginated task.list with the API key. The scraper also gained incremental fetching on updated_at, shape checks with quarantine, distinct exit codes for auth and upstream failures, a bounded run with a status file, and a nightly push. A deferred webhook item was closed as superseded, since the nightly poll keeps the archive current without a public endpoint.
THE RULE: A deferred migration should be triggered by the failure that actually matters, not only by the failure that is easiest to see. Any list call with a silent cap or no pagination can lose data while looking healthy, so the trigger must include checking completeness.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
1. Every list or discovery call in your pipeline is paginated, or you can state its documented maximum and show that your data stays below it.
2. A job that pulls records from an upstream source compares its count against an independent count or total, and fails or warns when they differ.
3. Each 'we will migrate when X breaks' note names a trigger that would actually fire for the known risk, such as silent truncation, and not only for a visible outage.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.
