benchmarks evals
The Model Cited the Law. Its Verdict Barely Moved.
A counterfactual benchmark audit tests whether a named authority actually influences a model’s legal conclusion.
Summary
A counterfactual benchmark audit tests whether a named authority actually influences a model’s legal conclusion.
Researchers held case facts fixed and replaced a model’s named legal authority with an unrelated one. Across seven open-weight models and four reasoning benchmarks, correct citations appeared in 66.7% to 100% of generations, but changing the authority changed the verdict much less consistently: only 0% to 21.7% on CaseHOLD, with higher and variable sensitivity elsewhere. A separate red-team test on five core models found 73.3% to 96.4% compliance with an adversarial instruction hidden in case facts. The authors used controls for prompt and sampling effects. These are controlled model tests, not judgments by a court. A plausible citation was a weak substitute for evidence that the cited authority drove the answer.
Why it matters
A counterfactual benchmark audit tests whether a named authority actually influences a model’s legal conclusion.
Limits and context
- Across seven open-weight models and four reasoning benchmarks, correct citations appeared in 66.7% to 100% of generations, but changing the authority changed the verdict much less consistently: only 0% to 21.7% on CaseHOLD, with higher and variable sensitivity elsewhere.
- These are controlled model tests, not judgments by a court.
Key claims
A counterfactual benchmark audit tests whether a named authority actually influences a model’s legal conclusion.
Qualification: Across seven open-weight models and four reasoning benchmarks, correct citations appeared in 66.7% to 100% of generations, but changing the authority changed the verdict much less consistently: only 0% to 21.7% on CaseHOLD, with higher and variable sensitivity elsewhere.
Evidence: source-2026-10-09-012
Sources
- arXiv preprint 2610.12361, original v1arXiv / submitting research authors · primary research
Corrections
No corrections have been recorded for this story.