TheMachine Press

A daily newspaper for the age of artificial intelligence.

Morning editionPermanent story

benchmarks evals

Disaster Vision Must Point to the Evidence

A 1.2-million-image benchmark links answers, captions and spatial relationships to visible regions across 142 countries.

Published Updated Story ID: mp-2026-11-10-002
Read the complete editionStory JSON

Summary

A 1.2-million-image benchmark links answers, captions and spatial relationships to visible regions across 142 countries.

DisasterVL is designed to test whether a visual model can justify what it says about a disaster scene rather than merely name a likely hazard. Its approximately 1.2 million images from 142 countries carry 25 million structured annotations spanning captions, incident and location assessments, spatial relations and visual questions. Answers are linked to visible evidence, including absent cues, while caption phrases and relationships point to image regions. The held-out set contains 38,254 images from 75 countries and 28 hazard types; Gemini 3.1 Pro audited its annotations and people validated a stratified subset. The authors report that their 9-billion-parameter DisasterVLM beats evaluated general-purpose baselines across captioning, open-ended questions, incident reasoning and region grounding, but text-overlap gains exceed evidence-score gains. The dataset, models and code are planned for release, so the current result remains a preprint benchmark claim awaiting broader independent use.

Why it matters

A 1.2-million-image benchmark links answers, captions and spatial relationships to visible regions across 142 countries.

Limits and context

No additional limitation was separately recorded.

Key claims

  1. A 1.2-million-image benchmark links answers, captions and spatial relationships to visible regions across 142 countries.

    Evidence: source-2026-11-10-002

Sources

  1. arXiv preprint 2611.12144, original v1arXiv · primary research

Corrections

No corrections have been recorded for this story.