infrastructure
Six GPUs Matched Seven After the Router Was Calibrated
A learned request router reached 0.864 mean goodput and matched round robin's seven-GPU result with six GPUs.
Summary
A learned request router reached 0.864 mean goodput and matched round robin's seven-GPU result with six GPUs.
Disaggregated serving separates prompt prefill from token decoding, making request placement a latency and capacity problem. This study predicts completion time from prompt length, expected output, post-admission cache pressure and service class, then calibrates the scorer on eight NVIDIA A40 GPUs. Across three bursty traces, mean goodput reached 0.864 versus 0.835 to 0.847 for round robin, least-loaded and length heuristics. Calibration mattered: simulator-only constants erased 4.5 goodput points and much of the tail-latency gain. Under extreme scarcity, greedy routing concentrated work too aggressively, marking a clear operating boundary.
Why it matters
A learned request router reached 0.864 mean goodput and matched round robin's seven-GPU result with six GPUs.
Limits and context
- Calibration mattered: simulator-only constants erased 4.5 goodput points and much of the tail-latency gain.
Key claims
A learned request router reached 0.864 mean goodput and matched round robin's seven-GPU result with six GPUs.
Qualification: Calibration mattered: simulator-only constants erased 4.5 goodput points and much of the tail-latency gain.
Evidence: source-2026-09-16-005
Sources
- arXiv preprint 2609.16206arXiv · primary research
Corrections
No corrections have been recorded for this story.