benchmarks evals
A Cheap Judge Escalated Its Uncertain Calls
Confidence routing preserved most comparator accuracy at lower reported cost.
Summary
Confidence routing preserved most comparator accuracy at lower reported cost.
A decision-only judge came within three points of a stronger comparator on ordinary preference and factuality tasks at 0.36% of its reported fee. A frozen confidence cascade retained 99% of comparator accuracy while escalating the harder cases, though derivations and polished wrong answers remained weak spots.
Why it matters
Confidence routing preserved most comparator accuracy at lower reported cost.
Limits and context
- A decision-only judge came within three points of a stronger comparator on ordinary preference and factuality tasks at 0.36% of its reported fee.
Key claims
Confidence routing preserved most comparator accuracy at lower reported cost.
Qualification: A decision-only judge came within three points of a stronger comparator on ordinary preference and factuality tasks at 0.36% of its reported fee.
Evidence: source-2026-09-23-020
Sources
- arXiv preprint 2609.26550arXiv · primary research
Corrections
No corrections have been recorded for this story.