benchmarks evals
A Correct Verdict Can Still Use the Wrong Patients
STRATA grades plans, evidence sets, p-values and verdicts for 8,735 temporal hypotheses drawn from vaccine adverse-event timelines.
Summary
STRATA grades plans, evidence sets, p-values and verdicts for 8,735 temporal hypotheses drawn from vaccine adverse-event timelines.
A fluent answer about whether one event follows another can hide a missing statistical test or the wrong evidence set. STRATA provides 8,735 temporal clinical hypotheses over vaccine adverse-event timelines, each with a gold plan, evidence set, p-value and verdict reproduced by a deterministic oracle. Across five model configurations, systems failed mainly when deciding which records counted, even when the final verdict looked correct. A staged articulate-retrieve-test workflow closed most of that gap and brought a 31-billion-parameter open model within 0.01 of frontier models under the same setup. The benchmark evaluates analysis procedure; it does not establish causal vaccine effects.
Why it matters
STRATA grades plans, evidence sets, p-values and verdicts for 8,735 temporal hypotheses drawn from vaccine adverse-event timelines.
Limits and context
- The benchmark evaluates analysis procedure; it does not establish causal vaccine effects.
Key claims
STRATA grades plans, evidence sets, p-values and verdicts for 8,735 temporal hypotheses drawn from vaccine adverse-event timelines.
Qualification: The benchmark evaluates analysis procedure; it does not establish causal vaccine effects.
Evidence: source-2026-11-01-007
Sources
- arXiv preprint 2610.40619, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.