research
The Reasoning Trace Works—Until the Problem Gets Harder
An RLVR model solved in-distribution equations with structured derivations, then abandoned that strategy for brute-force enumeration under harder shifts.
Summary
An RLVR model solved in-distribution equations with structured derivations, then abandoned that strategy for brute-force enumeration under harder shifts.
A controlled equation-solving setup makes strategy visible: a model must find multiple inputs for a non-bijective string function while tests vary the number of answers and output length. Reinforcement learning with verifiable rewards produced high in-distribution accuracy and orderly derivations, yet under harder out-of-distribution shifts the same model dropped the derivation and resorted to ineffective enumeration. The failure was neither a weak base model nor ordinary reward hacking. Co-training on bijective inversion tasks—different and not uniformly harder—made derivation more certain and restored much of the strategy and accuracy under shift. The finding is a case study, but it warns that outcome success can conceal brittle reasoning policies.
Why it matters
An RLVR model solved in-distribution equations with structured derivations, then abandoned that strategy for brute-force enumeration under harder shifts.
Limits and context
- Co-training on bijective inversion tasks—different and not uniformly harder—made derivation more certain and restored much of the strategy and accuracy under shift.
Key claims
An RLVR model solved in-distribution equations with structured derivations, then abandoned that strategy for brute-force enumeration under harder shifts.
Qualification: Co-training on bijective inversion tasks—different and not uniformly harder—made derivation more certain and restored much of the strategy and accuracy under shift.
Evidence: source-2026-11-03-010
Sources
- arXiv preprint 2611.03627, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.