policy regulation
Legal Models Still Choose When the Authority Is Missing
Generative systems selected a wrong holding in 38 to 86 percent of abstention cases; a decision model cut errors but did not eliminate them.
Summary
Generative systems selected a wrong holding in 38 to 86 percent of abstention cases; a decision model cut errors but did not eliminate them.
Researchers replaced the correct holding in 1,670 of 5,314 CaseHOLD items and asked models to recognize that none of the options applied. Generative language models still chose an answer in 38 to 86 percent of those cases, while the most cautious probability-based decision model did so in 30 percent. On ordinary CaseHOLD, one decision model matched the best generative system at a reported one fifty-fifth of the cost. Agreement filtering halved that model's wrong citations. The benchmark is a controlled legal-authority task, not legal advice or field validation.
Why it matters
Generative systems selected a wrong holding in 38 to 86 percent of abstention cases; a decision model cut errors but did not eliminate them.
Limits and context
- The benchmark is a controlled legal-authority task, not legal advice or field validation.
Key claims
Generative systems selected a wrong holding in 38 to 86 percent of abstention cases; a decision model cut errors but did not eliminate them.
Qualification: The benchmark is a controlled legal-authority task, not legal advice or field validation.
Evidence: source-2026-10-22-016
Sources
- arXiv preprint 2610.29961, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.