developer tools
Two Floating-Point Bugs Distort the Familiar Loss
Revised softmax and cross-entropy implementations preserve zero-sum gradients and exact teacher-student agreement in the tested settings.
Summary
Revised softmax and cross-entropy implementations preserve zero-sum gradients and exact teacher-student agreement in the tested settings.
The paper identifies collapse when one target logit dominates and cancellation when student and teacher logits nearly match. Its implementations remove both effects, changing grokking, distillation drift and post-memorization behavior in the authors’ experiments. The result concerns numerical implementations, not the mathematical definitions themselves.
Why it matters
Revised softmax and cross-entropy implementations preserve zero-sum gradients and exact teacher-student agreement in the tested settings.
Limits and context
- The result concerns numerical implementations, not the mathematical definitions themselves.
Key claims
Revised softmax and cross-entropy implementations preserve zero-sum gradients and exact teacher-student agreement in the tested settings.
Qualification: The result concerns numerical implementations, not the mathematical definitions themselves.
Evidence: source-2026-10-24-018
Sources
- arXiv preprint 2610.31642, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.