chips infrastructure
Four-Bit KV Cache Reaches the Tensor Core
Algorithm-system co-design keeps reported accuracy near BF16 while raising peak serving throughput 2.4 to 3 times on B300 hardware.
Summary
Algorithm-system co-design keeps reported accuracy near BF16 while raising peak serving throughput 2.4 to 3 times on B300 hardware.
BitDecoding-FP4 combines error controls for queries, probabilities and keys with fused paged-cache kernels. Across the authors’ models and NVIDIA B300 setup, NVFP4 stayed within 0.5 accuracy points of BF16 KV and delivered 2.4-to-3-times higher peak serving throughput.
Why it matters
Algorithm-system co-design keeps reported accuracy near BF16 while raising peak serving throughput 2.4 to 3 times on B300 hardware.
Limits and context
No additional limitation was separately recorded.
Key claims
Algorithm-system co-design keeps reported accuracy near BF16 while raising peak serving throughput 2.4 to 3 times on B300 hardware.
Evidence: source-2026-11-09-017
Sources
- arXiv preprint 2611.09825, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.