benchmarks evals
The Answer Was Right. The Time Series Was Ignored.
Interventions showed that aggregate accuracy could hide whether a QA system used the numerical evidence it was given.
Summary
Interventions showed that aggregate accuracy could hide whether a QA system used the numerical evidence it was given.
Researchers evaluated four time-series question-answering systems under the original input and six interventions that changed the supplied numerical series while holding the question and target fixed. Similar task-level scores often concealed substantial changes in individual predictions, and some systems fell back to simple behaviors rather than preserving the claimed capability. An audit of rationales also found numerical claims unsupported by the input and intermediate inferences that were invalid even when the explanation agreed with the final answer. The benchmark's warning is broader than time series: correct outputs do not establish that a system read, used or faithfully described its evidence.
Why it matters
Interventions showed that aggregate accuracy could hide whether a QA system used the numerical evidence it was given.
Limits and context
- The benchmark's warning is broader than time series: correct outputs do not establish that a system read, used or faithfully described its evidence.
Key claims
Interventions showed that aggregate accuracy could hide whether a QA system used the numerical evidence it was given.
Qualification: The benchmark's warning is broader than time series: correct outputs do not establish that a system read, used or faithfully described its evidence.
Evidence: source-2026-10-06-008
Sources
- arXiv preprint 2610.05686arXiv · primary research
Corrections
No corrections have been recorded for this story.