infrastructure
The Draft Model Stops Carrying the Whole Conversation
DLite reuses target-model hidden states for local speculative prediction instead of maintaining its own growing historical cache.
Summary
DLite reuses target-model hidden states for local speculative prediction instead of maintaining its own growing historical cache.
Long agent sessions can erode speculative-decoding acceptance while making a draft model's cache expensive. DLite performs local prediction from target-model hidden states without explicitly preserving the full history. On the authors' general, long-context and agentic tests, it produced acceptance lengths of two to six and outperformed the compared DSpark method by as much as 70 percent. The gains depend on the tested models and serving stack.
Why it matters
DLite reuses target-model hidden states for local speculative prediction instead of maintaining its own growing historical cache.
Limits and context
No additional limitation was separately recorded.
Key claims
DLite reuses target-model hidden states for local speculative prediction instead of maintaining its own growing historical cache.
Evidence: source-2026-10-19-018
Sources
- arXiv preprint 2610.23787, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.