benchmarks evals
Fifty Multimodal Models Still Lose the Moment and the Place
OmniCompass tests when, where and what happens in video under visual-only and audio-visual conditions.
Summary
OmniCompass tests when, where and what happens in video under visual-only and audio-visual conditions.
OmniCompass separates temporal grounding, spatial grounding and question answering across a visual-language track and an audio-visual track. The main collection contains 820 videos and 2,460 instances; a narrative and audio-rich subset contains 240 videos and 720 instances. Each target appears in paired text-only and text-plus-image specifications. Evaluations of 50 multimodal models found persistent temporal and spatial errors and inconsistencies between those paired prompts; the authors also report an open-source gap on audio-visual understanding. The benchmark diagnoses multiple dimensions, but its scores do not establish general video understanding outside the curated domains.
Why it matters
OmniCompass tests when, where and what happens in video under visual-only and audio-visual conditions.
Limits and context
- Each target appears in paired text-only and text-plus-image specifications.
- The benchmark diagnoses multiple dimensions, but its scores do not establish general video understanding outside the curated domains.
Key claims
OmniCompass tests when, where and what happens in video under visual-only and audio-visual conditions.
Qualification: Each target appears in paired text-only and text-plus-image specifications.
Evidence: source-2026-10-14-015
Sources
- arXiv preprint 2610.18829, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.