robotics
One Word Sent the Robot From Perfect to Almost Never
A vision-language-action policy switched on a stove in every trial under one instruction and only 2% under a near-synonym; a one-pass rewriter recovered part of the gap without retraining.
Summary
A vision-language-action policy switched on a stove in every trial under one instruction and only 2% under a near-synonym; a one-pass rewriter recovered part of the gap without retraining.
Language robustness in a vision model does not guarantee language robustness after that model is connected to action. Researchers measured statistically significant single-edit swings in vision-language-action policies: one tested policy completed the same LIBERO stove task 100% of the time for “switch on the stove” and 2% for “switch on the hot plate,” while a rephrase-augmented checkpoint still showed swings as large as 61 points. An oracle search suggested that phrasing alone nearly closed a 21-point in-distribution gap. The proposed mitigation scores alternative instructions on a few training tasks, distills the evidence into rewriting rules and rewrites each incoming command once before the frozen policy acts. Across twelve held-out tasks, the rules improved the tested policy by 16% to 27% relative. This is benchmark evidence, not a general safety guarantee, but it turns wording into an observable control variable rather than user error.
Why it matters
A vision-language-action policy switched on a stove in every trial under one instruction and only 2% under a near-synonym; a one-pass rewriter recovered part of the gap without retraining.
Limits and context
- Language robustness in a vision model does not guarantee language robustness after that model is connected to action.
- This is benchmark evidence, not a general safety guarantee, but it turns wording into an observable control variable rather than user error.
Key claims
A vision-language-action policy switched on a stove in every trial under one instruction and only 2% under a near-synonym; a one-pass rewriter recovered part of the gap without retraining.
Qualification: Language robustness in a vision model does not guarantee language robustness after that model is connected to action.
Evidence: source-2026-10-08-001
Sources
- arXiv preprint 2610.10526arXiv · primary research
Corrections
No corrections have been recorded for this story.