frontier models
One Vision Block Reused Itself at Different Depths
reViT matched a full-depth encoder with about 70% fewer stored parameters, while keeping inference computation comparable.
Summary
reViT matched a full-depth encoder with about 70% fewer stored parameters, while keeping inference computation comparable.
Reusing a Transformer block saves weights but can erase the transformations associated with different depths. reViT restores that variation by blending a small shared bank of feed-forward experts according to a continuous depth coordinate. Trained from scratch, reViT-B/16 matched DeiT III accuracy with about 70% fewer stored parameters at comparable inference FLOPs. An eight-expert model distilled from DINOv2 output features retained nearly all of its teacher’s linear-probe accuracy and transferred to segmentation and depth prediction. One checkpoint can run at several tested depths. Materializing it as a conventional dense graph removes online merging but expands storage. The reported saving is parameter storage, not a 70% reduction in inference work.
Why it matters
reViT matched a full-depth encoder with about 70% fewer stored parameters, while keeping inference computation comparable.
Limits and context
- reViT restores that variation by blending a small shared bank of feed-forward experts according to a continuous depth coordinate.
- The reported saving is parameter storage, not a 70% reduction in inference work.
Key claims
reViT matched a full-depth encoder with about 70% fewer stored parameters, while keeping inference computation comparable.
Qualification: reViT restores that variation by blending a small shared bank of feed-forward experts according to a continuous depth coordinate.
Evidence: source-2026-10-09-004
Sources
- arXiv preprint 2610.12448, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.