benchmarks evals
Physics Agents Know More Than They Can Prove
FrontierPhysics asks agents to complete 56 end-to-end research tasks authored by 42 physicists, then scores correctness and publishability.
Summary
FrontierPhysics asks agents to complete 56 end-to-end research tasks authored by 42 physicists, then scores correctness and publishability.
Each task spans literature review, planning, analysis or experiment, structured output and a scientific manuscript. The benchmark covers nine theoretical, computational and experimental domains and combines deterministic checks with expert-written rubrics. Across 22 models in 26 configurations, no configuration passes more than 46 of 168 rollouts. Manual trajectory review attributes the largest deficits to rigor and expert intuition rather than missing physics knowledge or tool execution. The benchmark measures the supplied research tasks, not autonomous discovery in every scientific field.
Why it matters
FrontierPhysics asks agents to complete 56 end-to-end research tasks authored by 42 physicists, then scores correctness and publishability.
Limits and context
- The benchmark measures the supplied research tasks, not autonomous discovery in every scientific field.
Key claims
FrontierPhysics asks agents to complete 56 end-to-end research tasks authored by 42 physicists, then scores correctness and publishability.
Qualification: The benchmark measures the supplied research tasks, not autonomous discovery in every scientific field.
Evidence: source-2026-11-05-008
Sources
- arXiv preprint 2611.06910, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.