chips infrastructure
Four-Bit AdamW Changed Where It Rounded
Two optimizer-state recipes reduced the validation-loss gap by changing how second-moment quantization affects the next update.
Summary
Two optimizer-state recipes reduced the validation-loss gap by changing how second-moment quantization affects the next update.
Small average error in an optimizer’s stored state need not mean small error in its adaptive update. This AdamW study moves stochastic-rounding decisions into preconditioner space with ZIP-SR, while a second recipe, ZE-EDEN, uses a zero-excluding codebook and rescaling to limit distortion. Both keep four-bit NormalFloat for the first moment. Across GPT- and Llama-style pretraining from 130 million to 2.7 billion parameters, both reduced TorchAO four-bit AdamW’s mean validation-loss gap against the 32-bit optimizer at every tested size. The largest reported gap reduction was 70%; that is a reduction in the gap, not in total loss. Full-parameter fine-tuning also improved over the tested four-bit baseline.
Why it matters
Two optimizer-state recipes reduced the validation-loss gap by changing how second-moment quantization affects the next update.
Limits and context
- Small average error in an optimizer’s stored state need not mean small error in its adaptive update.
- The largest reported gap reduction was 70%; that is a reduction in the gap, not in total loss.
Key claims
Two optimizer-state recipes reduced the validation-loss gap by changing how second-moment quantization affects the next update.
Qualification: Small average error in an optimizer’s stored state need not mean small error in its adaptive update.
Evidence: source-2026-10-09-008
Sources
- arXiv preprint 2610.12444, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.