chips infrastructure
Pruning Followed the Tokens the Model Actually Generated
SparseDecoding calibrates on autoregressive activations and adds a sparse matrix-vector kernel.
Summary
SparseDecoding calibrates on autoregressive activations and adds a sparse matrix-vector kernel.
Natural-text prefill and self-generated decoding can have different activation distributions. Calibrating on generation, excluding prefill, improved the tested pruned models’ long-form results. An optimized N:M kernel delivered up to 1.48 times end-to-end decoding speedup on A100 GPUs across the evaluated model families.
Why it matters
SparseDecoding calibrates on autoregressive activations and adds a sparse matrix-vector kernel.
Limits and context
No additional limitation was separately recorded.
Key claims
SparseDecoding calibrates on autoregressive activations and adds a sparse matrix-vector kernel.
Evidence: source-2026-10-09-019
Sources
- arXiv preprint 2610.12327, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.