frontier models
In-Context Learning Becomes a Key-Value Lookup
A causal circuit analysis traces arbitrary-label few-shot classification through label tokens, matching heads, a repeat gate and recency suppression.
Summary
A causal circuit analysis traces arbitrary-label few-shot classification through label tokens, matching heads, a repeat gate and recency suppression.
Across three transformers with three to four billion parameters, the authors find that middle-layer heads write a demonstration's class into the residual stream at its label token. Upper-layer heads later match the query class against those stored keys and copy the corresponding label. An early signal gates the readout when labels repeat, while another head set suppresses the most recent label. Activation patching, ablations and steering support each step, and related parts appear from 0.6 to 8 billion parameters in one family. The traced circuit covers a specific sentence-classification setting; it is not a general explanation of every form of in-context learning.
Why it matters
A causal circuit analysis traces arbitrary-label few-shot classification through label tokens, matching heads, a repeat gate and recency suppression.
Limits and context
- The traced circuit covers a specific sentence-classification setting; it is not a general explanation of every form of in-context learning.
Key claims
A causal circuit analysis traces arbitrary-label few-shot classification through label tokens, matching heads, a repeat gate and recency suppression.
Qualification: The traced circuit covers a specific sentence-classification setting; it is not a general explanation of every form of in-context learning.
Evidence: source-2026-11-04-008
Sources
- arXiv preprint 2611.05344, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.