benchmarks evals
The Best Vision Model Solves Three Quarters of the Planning Test
Plan-Bench isolates spatial, temporal and constraint reasoning from the execution of physical actions.
Summary
Plan-Bench isolates spatial, temporal and constraint reasoning from the execution of physical actions.
Plan-Bench asks whether vision-language models can build multi-step plans for embodied worlds without mixing in failures from the robot controller. Its 699 curated samples span 16 subclasses across spatial information, temporal state and task constraints. Among 28 evaluated models, the strongest reached 75.0 percent accuracy. The paper reports that larger models generally did better, but a quarter of the test still remained unsolved by the leader. Because execution is deliberately removed, the benchmark measures high-level planning rather than end-to-end robot competence.
Why it matters
Plan-Bench isolates spatial, temporal and constraint reasoning from the execution of physical actions.
Limits and context
No additional limitation was separately recorded.
Key claims
Plan-Bench isolates spatial, temporal and constraint reasoning from the execution of physical actions.
Evidence: source-2026-10-14-013
Sources
- arXiv preprint 2610.18836, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.