infrastructure
KV Cache Eviction Became an Estimation Problem
A probabilistic formulation exposes the bias in common eviction heuristics and enables decode-time correction.
Summary
A probabilistic formulation exposes the bias in common eviction heuristics and enables decode-time correction.
The paper formalizes optimal KV-cache eviction, proves the general problem computationally hard and recasts practical eviction as expectation estimation. From that view, existing methods become zero-variance biased estimators that can be modified to correct for discarded entries during decoding. The probabilistic variants were more robust across tasks at the same compression budget in the reported experiments, though the formulation does not remove the underlying computational hardness.
Why it matters
A probabilistic formulation exposes the bias in common eviction heuristics and enables decode-time correction.
Limits and context
- The probabilistic variants were more robust across tasks at the same compression budget in the reported experiments, though the formulation does not remove the underlying computational hardness.
Key claims
A probabilistic formulation exposes the bias in common eviction heuristics and enables decode-time correction.
Qualification: The probabilistic variants were more robust across tasks at the same compression budget in the reported experiments, though the formulation does not remove the underlying computational hardness.
Evidence: source-2026-08-31-016
Sources
- arXiv preprint 2608.28293arXiv · primary research
Corrections
No corrections have been recorded for this story.