benchmarks evals
The Model Can Calculate Correctly and Still Miss the Wrong Worksheet
MedPhysBench separates medical-physics calculation from error detection, exposing uneven performance on clinical tasks and planted mistakes.
Summary
MedPhysBench separates medical-physics calculation from error detection, exposing uneven performance on clinical tasks and planted mistakes.
MedPhysBench generates calculation and checking items across radiation therapy, imaging, nuclear medicine and radiation protection. On 1,200 held-out items, 12 models ranged from 33.3 to 100 percent calculation accuracy; the authors also found problems in two shielding items and disclose them. Five models caught at least 96.7 percent of mostly large planted errors with no more than 4 percent false alarms. In a targeted inverse-square test, some weaker models performed the correction in their own work yet accepted most worksheets that omitted it. The benchmark is a research evaluation with synthetic and public cases, not authorization for unsupervised clinical use.
Why it matters
MedPhysBench separates medical-physics calculation from error detection, exposing uneven performance on clinical tasks and planted mistakes.
Limits and context
- The benchmark is a research evaluation with synthetic and public cases, not authorization for unsupervised clinical use.
Key claims
MedPhysBench separates medical-physics calculation from error detection, exposing uneven performance on clinical tasks and planted mistakes.
Qualification: The benchmark is a research evaluation with synthetic and public cases, not authorization for unsupervised clinical use.
Evidence: source-2026-11-04-010
Sources
- arXiv preprint 2611.05340, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.