frontier models
Shared Experts Can Quiet the Router Too Much
MirrorMoE suppresses unnecessary experts per token to recover exploration lost when shared experts flatten routing signals.
Summary
MirrorMoE suppresses unnecessary experts per token to recover exploration lost when shared experts flatten routing signals.
Fine-grained mixture-of-experts models offer many possible expert combinations, but shared experts can reduce variation in token representations and make the router explore less of that larger space. MirrorMoE addresses the mismatch by shrinking the useful search space rather than simply adding noise: it suppresses unnecessary experts for each token so the router can focus on plausible combinations. The authors report consistent performance improvements across tested model scales. The abstract does not quantify every gain or establish that shared experts are broadly harmful; the contribution is a diagnosis of how two common design choices can work against one another.
Why it matters
MirrorMoE suppresses unnecessary experts per token to recover exploration lost when shared experts flatten routing signals.
Limits and context
- The abstract does not quantify every gain or establish that shared experts are broadly harmful; the contribution is a diagnosis of how two common design choices can work against one another.
Key claims
MirrorMoE suppresses unnecessary experts per token to recover exploration lost when shared experts flatten routing signals.
Qualification: The abstract does not quantify every gain or establish that shared experts are broadly harmful; the contribution is a diagnosis of how two common design choices can work against one another.
Evidence: source-2026-10-18-006
Sources
- arXiv preprint 2610.22353, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.