infrastructure
Two-Bit Residuals Cut Looped-Transformer Cache Storage
ResidualQuant used final-loop states as a reference and encoded similar earlier loops as low-precision differences.
Summary
ResidualQuant used final-loop states as a reference and encoded similar earlier loops as low-precision differences.
Mixed-precision tests retained accuracy near BF16 while reducing theoretical KV storage 80.7%. On an RTX 5090, lower memory traffic improved fixed-batch decode throughput up to 2.73 times and the larger feasible batch raised peak throughput up to 4.15 times in the reported models.
Why it matters
ResidualQuant used final-loop states as a reference and encoded similar earlier loops as low-precision differences.
Limits and context
No additional limitation was separately recorded.
Key claims
ResidualQuant used final-loop states as a reference and encoded similar earlier loops as low-precision differences.
Evidence: source-2026-10-08-019
Sources
- arXiv preprint 2610.10381arXiv · primary research
Corrections
No corrections have been recorded for this story.