chips infrastructure
The Slowest GPU Was Solving Two Different Problems
TEMPO models expert serving as memory-bound below one token threshold and compute-bound above it, then dispatches for the actual batch regime.
Summary
TEMPO models expert serving as memory-bound below one token threshold and compute-bound above it, then dispatches for the actual batch regime.
Measurements on two data-center GPU generations show that balancing token counts alone misses a regime change: small expert loads pay mainly for weight streaming, while larger loads pay for padded matrix-multiplication tiles. TEMPO solves a fixed-charge makespan problem outside the critical path and integrates with SGLang. On Qwen3-235B in the predicted win region, throughput rose four to six percent and p99 latency fell about 15.6 percent; a communication-bound DeepSeek-V3 case did not improve, matching the authors' phase diagram rather than a universal-win claim.
Why it matters
TEMPO models expert serving as memory-bound below one token threshold and compute-bound above it, then dispatches for the actual batch regime.
Limits and context
- On Qwen3-235B in the predicted win region, throughput rose four to six percent and p99 latency fell about 15.6 percent; a communication-bound DeepSeek-V3 case did not improve, matching the authors' phase diagram rather than a universal-win claim.
Key claims
TEMPO models expert serving as memory-bound below one token threshold and compute-bound above it, then dispatches for the actual batch regime.
Qualification: On Qwen3-235B in the predicted win region, throughput rose four to six percent and p99 latency fell about 15.6 percent; a communication-bound DeepSeek-V3 case did not improve, matching the authors' phase diagram rather than a universal-win claim.
Evidence: source-2026-08-14-008
Sources
- arXiv preprint 2608.13057arXiv · primary research
Corrections
No corrections have been recorded for this story.