frontier models
More Pretraining Reverses the Optimizer Winner
Muon minimizes pretraining loss faster, but AdamW overtakes it after fine-tuning once the pretraining data budget grows large enough.
Summary
Muon minimizes pretraining loss faster, but AdamW overtakes it after fine-tuning once the pretraining data budget grows large enough.
Muon has gained attention as a faster pretraining optimizer, but lower pretraining loss does not guarantee the strongest fine-tuned model. Across the paper's language-model experiments, AdamW moves ahead after fine-tuning beyond a sufficiently large data budget, and that crossover arrives at a smaller token-to-parameter ratio as model size increases. The authors propose a margin-to-spread ratio—correct-logit margin relative to the dispersion of wrong logits—to explain the reversal. Muon leads on that measure at small budgets and AdamW at large ones, matching the downstream pattern; a linear associative-memory analysis reproduces the direction after an update. The crossover depends on the tested regimes and is not a universal optimizer ranking.
Why it matters
Muon minimizes pretraining loss faster, but AdamW overtakes it after fine-tuning once the pretraining data budget grows large enough.
Limits and context
- Muon has gained attention as a faster pretraining optimizer, but lower pretraining loss does not guarantee the strongest fine-tuned model.
- The crossover depends on the tested regimes and is not a universal optimizer ranking.
Key claims
Muon minimizes pretraining loss faster, but AdamW overtakes it after fine-tuning once the pretraining data budget grows large enough.
Qualification: Muon has gained attention as a faster pretraining optimizer, but lower pretraining loss does not guarantee the strongest fine-tuned model.
Evidence: source-2026-11-03-009
Sources
- arXiv preprint 2611.03633, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.