benchmarks evals
The Best Agent Reproduced Only 14 Percent of the Papers
PAPERBENCHX replays submissions from scratch across 93 expert-built tasks and 12 scientific areas, scoring regenerated evidence rather than plausible-looking outputs.
Summary
PAPERBENCHX replays submissions from scratch across 93 expert-built tasks and 12 scientific areas, scoring regenerated evidence rather than plausible-looking outputs.
Before an AI system discovers new science, it should be able to reproduce established results. PAPERBENCHX spans 93 expert-constructed tasks in 12 areas, including electromagnetics, photonics, chemistry, materials, biology and robotics. Agents receive a paper, a task specification and a domain-native software environment, then must rebuild an experiment and produce a runnable workflow. Every submission is replayed from scratch and only regenerated evidence is scored. Across 10 frontier-agent configurations, the best fully reproduced 14 percent of tasks at the 90 percent threshold; mean rubric scores ranged from 27.53 to 59.08 percent. The benchmark measures computational reproduction, not all of science, but shows how partial progress can look much stronger than end-to-end evidence.
Why it matters
PAPERBENCHX replays submissions from scratch across 93 expert-built tasks and 12 scientific areas, scoring regenerated evidence rather than plausible-looking outputs.
Limits and context
- Every submission is replayed from scratch and only regenerated evidence is scored.
- The benchmark measures computational reproduction, not all of science, but shows how partial progress can look much stronger than end-to-end evidence.
Key claims
PAPERBENCHX replays submissions from scratch across 93 expert-built tasks and 12 scientific areas, scoring regenerated evidence rather than plausible-looking outputs.
Qualification: Every submission is replayed from scratch and only regenerated evidence is scored.
Evidence: source-2026-10-30-008
Sources
- arXiv preprint 2610.39744, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.