infrastructure
The Scheduler Cannot Predict Away Sampling Noise
Prefill-side decode-length predictors converged near the same bound once model, data and sampling conditions were fixed.
Summary
Prefill-side decode-length predictors converged near the same bound once model, data and sampling conditions were fixed.
Inference schedulers would benefit from knowing how long a response will be before generation starts. This analysis treats decoding as a stopped stochastic process and argues that autoregressive sampling places a floor under any predictor that sees only prefill hidden states. Across several predictor designs, results under fixed model, dataset and sampling settings converged near the derived bound, suggesting system randomness dominated model-design differences. The authors propose benchmarking with fixed generation conditions and relative-error metrics; the paper derives and tests limits rather than a universal production scheduler.
Why it matters
Prefill-side decode-length predictors converged near the same bound once model, data and sampling conditions were fixed.
Limits and context
- This analysis treats decoding as a stopped stochastic process and argues that autoregressive sampling places a floor under any predictor that sees only prefill hidden states.
Key claims
Prefill-side decode-length predictors converged near the same bound once model, data and sampling conditions were fixed.
Qualification: This analysis treats decoding as a stopped stochastic process and argues that autoregressive sampling places a floor under any predictor that sees only prefill hidden states.
Evidence: source-2026-10-31-005
Sources
- arXiv preprint 2610.40541, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.