infrastructure
Greedy Decoding Changed With Numeric Precision
BF16 and FP16 frequently produced different outputs from the same greedy run.
Summary
BF16 and FP16 frequently produced different outputs from the same greedy run.
The same prompts and greedy algorithm diverged between BF16 and FP16 on identical hardware for 49% to 100% of prompts across the reported evaluations. Selective FP32 recomputation at the output head improved agreement at low batch sizes but did not provide universal determinism.
Why it matters
BF16 and FP16 frequently produced different outputs from the same greedy run.
Limits and context
- Selective FP32 recomputation at the output head improved agreement at low batch sizes but did not provide universal determinism.
Key claims
BF16 and FP16 frequently produced different outputs from the same greedy run.
Qualification: Selective FP32 recomputation at the output head improved agreement at low batch sizes but did not provide universal determinism.
Evidence: source-2026-09-23-019
Sources
- arXiv preprint 2609.26621arXiv · primary research
Corrections
No corrections have been recorded for this story.