benchmarks evals
The Benchmark Stopped Sampling What It Already Knew
A Bayesian stopping rule removed 57 to 97 percent of planned trials in illustrative evaluations while preserving the overall conclusion.
Summary
A Bayesian stopping rule removed 57 to 97 percent of planned trials in illustrative evaluations while preserving the overall conclusion.
Optstop treats evaluation as sequential measurement instead of assigning every item the same fixed number of trials. It keeps uncertain items eligible, stops when estimates are precise or stable, and becomes more cautious near zero performance where rare successes matter. Across nine validation settings in an illustrative 200-item, ten-epoch evaluation, it removed 57 to 97 percent of planned trials; realized savings depend on the benchmark and stopping target.
Why it matters
A Bayesian stopping rule removed 57 to 97 percent of planned trials in illustrative evaluations while preserving the overall conclusion.
Limits and context
No additional limitation was separately recorded.
Key claims
A Bayesian stopping rule removed 57 to 97 percent of planned trials in illustrative evaluations while preserving the overall conclusion.
Evidence: source-2026-08-17-007
Sources
- arXiv preprint 2608.14425arXiv · primary research
Corrections
No corrections have been recorded for this story.