benchmarks evals
Thirty Percent of the Benchmark Needed Repair
A continuous-validation audit turns Terminal-Bench 2.0 into 2.1 and shows that task defects and time budgets both move results.
Summary
A continuous-validation audit turns Terminal-Bench 2.0 into 2.1 and shows that task defects and time budgets both move results.
The workflow combines community reports, task artifacts, traces, automated oracle checks and human-gated repairs. The audit finds issues in roughly 30 percent of Terminal-Bench 2.0; repairs raise accuracy by more than 10 percentage points and move leaderboard positions by up to three ranks, though rank correlation remains high. Extending wall-clock limits also closes large harness gaps for two tested models, showing that resource budgets belong in the benchmark specification.
Why it matters
A continuous-validation audit turns Terminal-Bench 2.0 into 2.1 and shows that task defects and time budgets both move results.
Limits and context
No additional limitation was separately recorded.
Key claims
A continuous-validation audit turns Terminal-Bench 2.0 into 2.1 and shows that task defects and time budgets both move results.
Evidence: source-2026-11-06-009
Sources
- arXiv preprint 2611.08417, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.