research
The Biology Agent Can Find the Protein but Misread Its Shape
ProteinGeoBench separates metadata lookup from numerical geometry and shows how harness, access and run budget can change the ranking.
Summary
ProteinGeoBench separates metadata lookup from numerical geometry and shows how harness, access and run budget can change the ranking.
Structural-biology agents are often asked to retrieve facts and compute geometry in one score. ProteinGeoBench splits those abilities across 131 tasks. In 4,249 rollouts from four frontier models, three landed within 0.031 of one another; three of the four performed better on metadata lookup than numerical calculation. Numeric questions made up 28 tasks but half of the twelve every model failed. Rankings also changed with run budget and command-line harness. The study does not crown one universal agent; it argues evaluations should expose task family, repeats, access and harness instead of letting lookup fluency hide geometric weakness.
Why it matters
ProteinGeoBench separates metadata lookup from numerical geometry and shows how harness, access and run budget can change the ranking.
Limits and context
- The study does not crown one universal agent; it argues evaluations should expose task family, repeats, access and harness instead of letting lookup fluency hide geometric weakness.
Key claims
ProteinGeoBench separates metadata lookup from numerical geometry and shows how harness, access and run budget can change the ranking.
Qualification: The study does not crown one universal agent; it argues evaluations should expose task family, repeats, access and harness instead of letting lookup fluency hide geometric weakness.
Evidence: source-2026-10-20-013
Sources
- arXiv preprint 2610.26829, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.