chips infrastructure
A Rejected Draft Can Turn an Energy Saving Into a Loss
Production-style H100 and H200 measurements found speculative decoding saved energy only when acceptance stayed high, with the benefit shrinking as batches grew.
Summary
Production-style H100 and H200 measurements found speculative decoding saved energy only when acceptance stayed high, with the benefit shrinking as batches grew.
Verification dominated dense-model energy, and large batches already amortized weight reads. At high batch size, low-acceptance or compute-bound drafting could raise energy per accepted token even while meeting latency targets.
Why it matters
Production-style H100 and H200 measurements found speculative decoding saved energy only when acceptance stayed high, with the benefit shrinking as batches grew.
Limits and context
No additional limitation was separately recorded.
Key claims
Production-style H100 and H200 measurements found speculative decoding saved energy only when acceptance stayed high, with the benefit shrinking as batches grew.
Evidence: source-2026-11-01-021
Sources
- arXiv preprint 2610.40287, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.