safety security
A Complete-Looking Crisis Report Can Still Miss the Authority
An evaluation of disaster situation reports finds that surface similarity does not measure provenance or official coverage.
Summary
An evaluation of disaster situation reports finds that surface similarity does not measure provenance or official coverage.
A new evaluation derives a 15-slot schema from professional humanitarian situation reports, then compares 121 human and 261 machine-generated reports across five disasters. On one Cyclone Idai setting, a Gemini-based score put the best machine arm nominally above the empirical human ceiling, but the difference was statistically indistinguishable from zero and did not replicate with another model. Across three primary events, machine reports reproduced only 5.4 to 11.0 percent of same-day official figures. Date, organization, series and authority were underrepresented in every observed event-judge cell. The results are tied to this corpus and judging design, but they warn against treating a fluent, complete-looking situation report as grounded operational evidence.
Why it matters
An evaluation of disaster situation reports finds that surface similarity does not measure provenance or official coverage.
Limits and context
- On one Cyclone Idai setting, a Gemini-based score put the best machine arm nominally above the empirical human ceiling, but the difference was statistically indistinguishable from zero and did not replicate with another model.
- Across three primary events, machine reports reproduced only 5.4 to 11.0 percent of same-day official figures.
Key claims
An evaluation of disaster situation reports finds that surface similarity does not measure provenance or official coverage.
Qualification: On one Cyclone Idai setting, a Gemini-based score put the best machine arm nominally above the empirical human ceiling, but the difference was statistically indistinguishable from zero and did not replicate with another model.
Evidence: source-2026-10-14-016
Sources
- arXiv preprint 2610.18762, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.