benchmarks evals
A High Score Hid an Impossible Causal Claim
CERTID tests whether a reasoning model knows when observational data cannot identify the effect it is asked to compute.
Summary
CERTID tests whether a reasoning model knows when observational data cannot identify the effect it is asked to compute.
A model can refuse a solvable causal query or confidently supply a formula for an effect that observational data cannot identify. CERTID certifies identifiability from the graph and query, then checks formulas against structural causal models with known interventional distributions. On 1,200 instances spanning four to fifty vertices, three tested frontier models showed a seventeen-fold difference in false claims on non-identifiable questions. Their overall accuracy did not summarize that risk. The preprint evaluates formal graph problems, not unrestricted real-world causal inference, but makes an unsupported answer distinguishable from an ordinary mistake.
Why it matters
CERTID tests whether a reasoning model knows when observational data cannot identify the effect it is asked to compute.
Limits and context
- A model can refuse a solvable causal query or confidently supply a formula for an effect that observational data cannot identify.
- Their overall accuracy did not summarize that risk.
- The preprint evaluates formal graph problems, not unrestricted real-world causal inference, but makes an unsupported answer distinguishable from an ordinary mistake.
Key claims
CERTID tests whether a reasoning model knows when observational data cannot identify the effect it is asked to compute.
Qualification: A model can refuse a solvable causal query or confidently supply a formula for an effect that observational data cannot identify.
Evidence: source-2026-10-04-016
Sources
- arXiv preprint 2610.03519, original v1arXiv / submitting research authors · primary research
Corrections
: Publication note: This October 4 edition was reconstructed on October 5 after the workstation was offline. It uses original October 2 submissions from the standard Sunday-evening arXiv release, rather than claiming those papers were available for the missed 9 AM run. The date and 9 AM publication slot preserve archive chronology; October 5 developments are excluded. The September 27 Invention Desk is carried over because its replacement was generated October 5.