infrastructure
One Transformer Slides Across Its Compute Budget
A retraining-free ranking of individual neurons and attention heads lets one pretrained model trade capacity for speed across a continuum of sparsity targets.
Summary
A retraining-free ranking of individual neurons and attention heads lets one pretrained model trade capacity for speed across a continuum of sparsity targets.
Model families normally force a fixed size choice even when hardware and load change. This method jointly ranks individual MLP neurons and attention heads while optimizing across many soft-pruning targets. Across 27 encoders from 22 million to 8 billion parameters, the authors report smooth degradation through 60 percent sparsity. At 50 percent sparsity, a DINOv3 encoder loses 427 million parameters with a 0.8-point linear-probe accuracy drop; another model reaches 1.80 times throughput at 60 percent sparsity. Those figures are workload-specific, but the output is one adjustable model rather than a shelf of separately trained sizes.
Why it matters
A retraining-free ranking of individual neurons and attention heads lets one pretrained model trade capacity for speed across a continuum of sparsity targets.
Limits and context
No additional limitation was separately recorded.
Key claims
A retraining-free ranking of individual neurons and attention heads lets one pretrained model trade capacity for speed across a continuum of sparsity targets.
Evidence: source-2026-10-15-008
Sources
- arXiv preprint 2610.20588, original v1arXiv / submitting research authors · primary research
Corrections
: Publication note: This October 15 edition was prepared before its scheduled 9:00 AM America/Detroit reveal, but publication was delayed by a production build failure. The original reporting and image-generation timestamps are preserved.