chips infrastructure
A Sparse Attention Rescue Cut First-Token Time
Radius-bounded selection recovered important tokens that block averages would miss.
Summary
Radius-bounded selection recovered important tokens that block averages would miss.
RBS-Attention addresses “mean dilution” in long-context prefill: a block of keys can appear unimportant on average while hiding a strongly relevant token. Its training-free selector combines a centroid branch with a radius-based rescue branch, then runs ordinary block-sparse FlashAttention. On H100 GPUs with a 128,000-token context and Qwen3-30B-A3B, the authors measured a 5.97-fold end-to-end reduction in time to first token against their comparison setup; standalone prefill attention sped up 20.65-fold. A dense Qwen3-32B comparison scored 88.65 on RULER versus 89.52 with dense attention. Those are reported benchmark results for specified models and hardware, not a general speed guarantee.
Why it matters
Radius-bounded selection recovered important tokens that block averages would miss.
Limits and context
- Those are reported benchmark results for specified models and hardware, not a general speed guarantee.
Key claims
Radius-bounded selection recovered important tokens that block averages would miss.
Qualification: Those are reported benchmark results for specified models and hardware, not a general speed guarantee.
Evidence: source-2026-09-21-002
Sources
- arXiv preprint 2609.20971arXiv · primary research
Corrections
No corrections have been recorded for this story.