frontier models
Novel Reasoning Was Explored Somewhere Else
Exploration-Distillation separated novelty-seeking policies from the student that ultimately learned their filtered successes.
Summary
Exploration-Distillation separated novelty-seeking policies from the student that ultimately learned their filtered successes.
Novelty bonuses can push reinforcement learning with verifiable rewards toward new solutions, but they can also damage behavior that sparse task rewards cannot restore. Exploration-Distillation trains separate explorer policies with novelty incentives, filters their trajectories for correctness and quality, and distills the survivors into a student trained without the bonus. Repeating that cycle lets exploration become aggressive without making the deployed student absorb every experiment. Across seven mathematics benchmarks and two model families, the authors report better results than DAPO at the same wall-clock budget and stronger pass-at-k scaling, indicating more diverse correct solutions. The paper's operational point is broader than math: exploration and optimization need not occur in the same policy at the same time.
Why it matters
Exploration-Distillation separated novelty-seeking policies from the student that ultimately learned their filtered successes.
Limits and context
- Novelty bonuses can push reinforcement learning with verifiable rewards toward new solutions, but they can also damage behavior that sparse task rewards cannot restore.
- The paper's operational point is broader than math: exploration and optimization need not occur in the same policy at the same time.
Key claims
Exploration-Distillation separated novelty-seeking policies from the student that ultimately learned their filtered successes.
Qualification: Novelty bonuses can push reinforcement learning with verifiable rewards toward new solutions, but they can also damage behavior that sparse task rewards cannot restore.
Evidence: source-2026-10-08-004
Sources
- arXiv preprint 2610.10536arXiv · primary research
Corrections
No corrections have been recorded for this story.