Million-Token Memory Without the Global Bottleneck

Exact top-k sparse attention can preserve quality over long contexts, but coordinating a global selection across many GPUs can erase the computational savings. In a July ICML workshop preprint, Microsoft researchers Yifan Guo, Wei Cui, and Peng Cheng propose Interleaved DeepSeek Sparse Attention. Tokens are distributed in an interleaved layout and each device performs a relaxed local selection; the union of those selections is designed to cover nearly all globally relevant tokens with less synchronization. This is a research proposal, not a deployed product claim, and its practical value depends on evaluation across models and hardware. Still, it reframes the long-context problem as a systems question: how to retain useful memory without making every processor wait for a perfect global vote.