benchmarks evals
One Object in Three Mirrors Still Counts as One
The strongest tested multimodal model answered every related question correctly in only 13.36 percent of PhysCount groups.
Summary
The strongest tested multimodal model answered every related question correctly in only 13.36 percent of PhysCount groups.
Visual counting usually treats each detected appearance as an object, but reflections and transparent surfaces can show the same physical entity several times. PhysCount pairs 546 real-world images with 4,595 human-written questions organized into 651 groups, annotating appearances, observation modes and entity identity. Across 40 multimodal models, appearance counting was easier than linking appearances back to physical objects. The strongest model completed every question in only 13.36 percent of groups. Higher resolution helped expose visual evidence but only modestly improved full-group accuracy, while answering related questions jointly helped more.
Why it matters
The strongest tested multimodal model answered every related question correctly in only 13.36 percent of PhysCount groups.
Limits and context
- The strongest model completed every question in only 13.36 percent of groups.
- Higher resolution helped expose visual evidence but only modestly improved full-group accuracy, while answering related questions jointly helped more.
Key claims
The strongest tested multimodal model answered every related question correctly in only 13.36 percent of PhysCount groups.
Qualification: The strongest model completed every question in only 13.36 percent of groups.
Evidence: source-2026-11-01-012
Sources
- arXiv preprint 2610.40530, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.