benchmarks evals
Scientific Software Exposed the Limits of Computer-Use Agents
OSWorld-Science judged generated molecules, masks, plots and numbers—not just screenshots or final clicks.
Summary
OSWorld-Science judged generated molecules, masks, plots and numbers—not just screenshots or final clicks.
OSWorld-Science provides 146 expert-designed tasks across molecular drawing, retrosynthesis, pathology imaging, statistics and physical simulation. Execution-based evaluators inspect application state and artifacts such as molecular structures, segmentation masks, plots and numerical results, awarding partial credit when work is incomplete. Tests of twelve vision-language models with a strong harness still found substantial difficulty. The benchmark connects agent action to scientific outputs, but it does not establish that its automated checks replace domain-expert review.
Why it matters
OSWorld-Science judged generated molecules, masks, plots and numbers—not just screenshots or final clicks.
Limits and context
- The benchmark connects agent action to scientific outputs, but it does not establish that its automated checks replace domain-expert review.
Key claims
OSWorld-Science judged generated molecules, masks, plots and numbers—not just screenshots or final clicks.
Qualification: The benchmark connects agent action to scientific outputs, but it does not establish that its automated checks replace domain-expert review.
Evidence: source-2026-10-01-015
Sources
- arXiv preprint 2609.39903arXiv · primary research
Corrections
No corrections have been recorded for this story.