benchmarks evals
Fifty Thousand Agent Errors Became Training Pairs
AED preserved traces, diagnoses and replay evidence instead of reducing each failed rollout to a zero reward.
Summary
AED preserved traces, diagnoses and replay evidence instead of reducing each failed rollout to a zero reward.
The Agent Error Dataset contains 50,228 error-diagnosis pairs drawn from 9,961 tasks, 33 environments, 19 harness families and 23 policy models. Its pipeline proposes corrections and, where replay is available, compares them with retries from the same checkpoint. Across 3,062 matched pairs, first-proposal corrections raised verifier pass rates from 18.4% to 51.1%. A separately frozen diagnosis set also improved an 8B model's agreement with teacher labels, though those labels remain part of the reported evaluation design.
Why it matters
AED preserved traces, diagnoses and replay evidence instead of reducing each failed rollout to a zero reward.
Limits and context
- A separately frozen diagnosis set also improved an 8B model's agreement with teacher labels, though those labels remain part of the reported evaluation design.
Key claims
AED preserved traces, diagnoses and replay evidence instead of reducing each failed rollout to a zero reward.
Qualification: A separately frozen diagnosis set also improved an 8B model's agreement with teacher labels, though those labels remain part of the reported evaluation design.
Evidence: source-2026-10-01-014
Sources
- arXiv preprint 2609.40111arXiv · primary research
Corrections
No corrections have been recorded for this story.