infrastructure
Each Expert Drops Its Gradient Early
Sonar-MoE updates expert tensors during backpropagation and frees their gradients immediately, cutting peak memory by 19 to 22 percent in the tested models.
Summary
Sonar-MoE updates expert tensors during backpropagation and frees their gradients immediately, cutting peak memory by 19 to 22 percent in the tested models.
The updates remain bitwise identical without expert clipping, while global-clipping runs improve step time by 17 to 23 percent. Batched expert updates avoid the slowdown of processing experts one by one.
Why it matters
Sonar-MoE updates expert tensors during backpropagation and frees their gradients immediately, cutting peak memory by 19 to 22 percent in the tested models.
Limits and context
- The updates remain bitwise identical without expert clipping, while global-clipping runs improve step time by 17 to 23 percent.
Key claims
Sonar-MoE updates expert tensors during backpropagation and frees their gradients immediately, cutting peak memory by 19 to 22 percent in the tested models.
Qualification: The updates remain bitwise identical without expert clipping, while global-clipping runs improve step time by 17 to 23 percent.
Evidence: source-2026-11-03-017
Sources
- arXiv preprint 2611.03577, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.