benchmarks evals
The Model Learned to Withhold Some Answers. The Guarantee Did Not Travel.
A separate answer selector improved one factual dataset but still lost to silence on another.
Summary
A separate answer selector improved one factual dataset but still lost to silence on another.
Researchers asked whether reliable abstention should come from training the language model itself or from a separate component that checks its proposed answers. On Qwen3-14B, the learned selector improved threshold-averaged utility over the original model’s calibrated confidence filter. Under exact-match grading on TriviaQA, mean coverage rose from 20.2% to 30.9%, while the pooled error rate among scored answers fell from 23.0% to 14.5%. The tested reinforcement-learning changes did not produce consistent gains. On NQ-Open, the selector improved on the filter but still performed worse than always abstaining. A partial human-reviewed audit preserved the TriviaQA finding. The contrast matters for deploying a system that must decide whether it knows enough: selecting answers can help in a particular setting, but one dataset’s improvement does not establish reliable uncertainty handling elsewhere. The authors leave that broader problem unresolved.
Why it matters
A separate answer selector improved one factual dataset but still lost to silence on another.
Limits and context
- The tested reinforcement-learning changes did not produce consistent gains.
- The contrast matters for deploying a system that must decide whether it knows enough: selecting answers can help in a particular setting, but one dataset’s improvement does not establish reliable uncertainty handling elsewhere.
Key claims
A separate answer selector improved one factual dataset but still lost to silence on another.
Qualification: The tested reinforcement-learning changes did not produce consistent gains.
Evidence: source-2026-10-12-016
Sources
- arXiv preprint 2610.13821, original v1arXiv / submitting research authors · primary research
Corrections
: Publication note: The October 12, 2026 edition was prepared and published late on October 12 after the scheduled newsroom task could not start. The edition retains its October 12 date and 9:00 AM America/Detroit reveal time. Reports use sources available before that cutoff; actual preparation and image-generation times are preserved in the provenance records.