benchmarks evals
The Failure Predictor Mostly Recognizes the Issue and Model
On public coding-agent trajectories, issue difficulty and model identity explain far more than the partial run itself.
Summary
On public coding-agent trajectories, issue difficulty and model identity explain far more than the partial run itself.
A run-independent issue-difficulty score reaches 0.92 AUROC on SWE-bench Verified trajectories, while trajectory predictors reach about 0.62. Within an issue, model identity predicts failure better than the studied predictors, and per-issue model switching provides no held-out gain. A repeated-run pilot shows that retries can rescue some failures, but current predictors cannot distinguish a bad run from a task beyond the model’s ability. The result argues for repeated-run, closed-loop evaluation rather than universal pessimism about monitoring.
Why it matters
On public coding-agent trajectories, issue difficulty and model identity explain far more than the partial run itself.
Limits and context
- A repeated-run pilot shows that retries can rescue some failures, but current predictors cannot distinguish a bad run from a task beyond the model’s ability.
Key claims
On public coding-agent trajectories, issue difficulty and model identity explain far more than the partial run itself.
Qualification: A repeated-run pilot shows that retries can rescue some failures, but current predictors cannot distinguish a bad run from a task beyond the model’s ability.
Evidence: source-2026-11-06-016
Sources
- arXiv preprint 2611.08285, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.