benchmarks evals
The Caption Looked Fluent While Identities Drifted
OmniCapBench replaces one whole-caption score with atomic checks for entities, shots and audio events.
Summary
OmniCapBench replaces one whole-caption score with atomic checks for entities, shots and audio events.
OmniCapBench decomposes audio-visual captions into verifiable units across entity references, visual shots and audio events. Its 786 densely annotated videos support deterministic constraint checks and localized semantic comparisons, exposing where a polished caption loses temporal grounding, swaps identities, misaligns sound and image, or invents details. Testing frontier multimodal models found strong local perception but weaker long-horizon reasoning, especially for identity continuity and cross-modal alignment. The benchmark is intended as a diagnostic map rather than a single leaderboard number. Its judgments still depend partly on model-based semantic comparison, but the atomic structure makes errors easier to locate and audit.
Why it matters
OmniCapBench replaces one whole-caption score with atomic checks for entities, shots and audio events.
Limits and context
No additional limitation was separately recorded.
Key claims
OmniCapBench replaces one whole-caption score with atomic checks for entities, shots and audio events.
Evidence: source-2026-10-10-010
Sources
- arXiv preprint 2610.12458, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.