benchmarks evals
The Lab Model Follows the Procedure but Misses the Place to Act
BEAKER tests 28 multimodal models on 1,000 expert-authored image and video questions for self-driving chemical laboratories.
Summary
BEAKER tests 28 multimodal models on 1,000 expert-authored image and video questions for self-driving chemical laboratories.
Eleven task types cover affordance grounding, trajectories, apparatus, state and safety recognition, setup, operation and temporal procedure reasoning. Chemistry and embodied-AI experts created and reviewed every item without AI-assisted annotation. Current models handle coarse procedures in video better than operable regions, feasible paths, fine-grained apparatus and safety norms, and mid-length step ordering. BEAKER is an offline, single-turn benchmark; it diagnoses gaps in laboratory actionability rather than demonstrating a robot running experiments.
Why it matters
BEAKER tests 28 multimodal models on 1,000 expert-authored image and video questions for self-driving chemical laboratories.
Limits and context
- BEAKER is an offline, single-turn benchmark; it diagnoses gaps in laboratory actionability rather than demonstrating a robot running experiments.
Key claims
BEAKER tests 28 multimodal models on 1,000 expert-authored image and video questions for self-driving chemical laboratories.
Qualification: BEAKER is an offline, single-turn benchmark; it diagnoses gaps in laboratory actionability rather than demonstrating a robot running experiments.
Evidence: source-2026-11-05-007
Sources
- arXiv preprint 2611.06911, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.