safety security
A Value's Name Predicted Less Than Its Activation
Model activations forecast how post-training on one value affected unseen values better than text descriptions did.
Summary
Model activations forecast how post-training on one value affected unseen values better than text descriptions did.
Researchers define alignment generalization prediction as forecasting how tuning for one value changes behavior across held-out values. Their analysis covers 66 values drawn from modern alignment targets. Representations built from model activations while values were applied in context reached a reported correlation of 0.45 with the measured generalization matrix, versus 0.05 for description-based baselines. The authors also connect similarity within a multi-value target to robustness and report early evidence for a shared value space across models. The results do not make value alignment predictable in general; they show that internal, contextual representations carried more information than labels in the tested setup.
Why it matters
Model activations forecast how post-training on one value affected unseen values better than text descriptions did.
Limits and context
- The results do not make value alignment predictable in general; they show that internal, contextual representations carried more information than labels in the tested setup.
Key claims
Model activations forecast how post-training on one value affected unseen values better than text descriptions did.
Qualification: The results do not make value alignment predictable in general; they show that internal, contextual representations carried more information than labels in the tested setup.
Evidence: source-2026-10-11-005
Sources
- arXiv preprint 2610.12410, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.