developer tools
Eight Steps Were Enough to Flag Some Failing Runs
OnTrack compares streaming agent structure with successful trajectories at about a millisecond per step.
Summary
OnTrack compares streaming agent structure with successful trajectories at about a millisecond per step.
On SWE-bench trajectories, the first eight steps improved failure ranking by 0.057 AUROC over content-similarity methods. A tested abort policy saved about 18% of compute destined for failing runs; five of six aborts were correct. Monitoring weakened as access to reference runs and schemas decreased.
Why it matters
OnTrack compares streaming agent structure with successful trajectories at about a millisecond per step.
Limits and context
No additional limitation was separately recorded.
Key claims
OnTrack compares streaming agent structure with successful trajectories at about a millisecond per step.
Evidence: source-2026-10-09-017
Sources
- arXiv preprint 2610.12375, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.