frontier models
Ten Million Tokens Get a Hierarchical Index
Latent Sparse Attention clusters keys in a learned hierarchy and reports a 72.4-times layer-level prefill speedup over full attention at ten million tokens.
Summary
Latent Sparse Attention clusters keys in a learned hierarchy and reports a 72.4-times layer-level prefill speedup over full attention at ten million tokens.
Sparse attention still struggles when its indexer either touches every key or loses accuracy at extreme context lengths. Latent Sparse Attention clusters keys hierarchically in a learned latent space and lets each query score clusters instead of all keys. A distilled version can retrofit pretrained models; a native version learns from next-token loss. On the reported GLM-5.3-FLASH test at ten million tokens, one layer ran 22.7 times faster in prefill than the model's original sparse attention and 72.4 times faster than full attention, with similar task accuracy; decode gains were 4.6 and 42.5 times respectively. These are layer-level results, not end-to-end application latency guarantees.
Why it matters
Latent Sparse Attention clusters keys in a learned hierarchy and reports a 72.4-times layer-level prefill speedup over full attention at ten million tokens.
Limits and context
- These are layer-level results, not end-to-end application latency guarantees.
Key claims
Latent Sparse Attention clusters keys in a learned hierarchy and reports a 72.4-times layer-level prefill speedup over full attention at ten million tokens.
Qualification: These are layer-level results, not end-to-end application latency guarantees.
Evidence: source-2026-11-02-005
Sources
- arXiv preprint 2610.42147, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.