benchmarks evals
A Ninety-Percent-Accurate Monitor Can Still Make Restarts Worse
The same monitor ranges from a five-point loss to a 35-point gain depending on the replacement attempt, and none of 103 live comparisons cleared the study's decision bar.
Summary
The same monitor ranges from a five-point loss to a 35-point gain depending on the replacement attempt, and none of 103 live comparisons cleared the study's decision bar.
Accuracy alone cannot say whether an alarm should restart an agent. An exact identity in this paper shows that a monitor fixed at 90 percent accuracy can move total success by anywhere from minus five to plus 35 percentage points depending only on the residual-budget success probability of the replacement attempt. The authors add a decision-sufficiency test for progress representations and a finite-sample audit that returns restart, continue or abstain. On 194 externally evaluated executions across 19 selected SymPy issues, an apparent 2.29-point gain traced to one surviving failed attempt, a second model lost 42.29 points, and none of 103 live comparisons met the stated certification threshold. The analysis is scoped to independent clean restarts with the declared assumptions.
Why it matters
The same monitor ranges from a five-point loss to a 35-point gain depending on the replacement attempt, and none of 103 live comparisons cleared the study's decision bar.
Limits and context
- Accuracy alone cannot say whether an alarm should restart an agent.
- An exact identity in this paper shows that a monitor fixed at 90 percent accuracy can move total success by anywhere from minus five to plus 35 percentage points depending only on the residual-budget success probability of the replacement attempt.
Key claims
The same monitor ranges from a five-point loss to a 35-point gain depending on the replacement attempt, and none of 103 live comparisons cleared the study's decision bar.
Qualification: Accuracy alone cannot say whether an alarm should restart an agent.
Evidence: source-2026-10-28-016
Sources
- arXiv preprint 2610.36544, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.