benchmarks evals
A Correct Answer Rarely Meant the Hallucination Was Repaired
PHRBench classified whether models complied with, avoided or heuristically corrected a false premise across 4,820 controlled cases.
Summary
PHRBench classified whether models complied with, avoided or heuristically corrected a false premise across 4,820 controlled cases.
Hallucinated context can become the premise for another model's reasoning, so final-answer accuracy alone cannot show how the error was handled. PHRBench evaluates eighteen language models across four domains and labels trajectories as compliance, avoidance or heuristic correction, reserving an insightful outcome for a correction that also reaches the right answer. Successful recovery was relatively rare and coincided with more belief updates through the trace. Prompt properties alone carried enough signal for a lightweight predictor to reach 0.847 AUROC on recovery. The benchmark does not remove hallucinations, but it makes downstream behavior inspectable: a system that lands on the right answer may still have accepted the false premise for the wrong reasons.
Why it matters
PHRBench classified whether models complied with, avoided or heuristically corrected a false premise across 4,820 controlled cases.
Limits and context
- Hallucinated context can become the premise for another model's reasoning, so final-answer accuracy alone cannot show how the error was handled.
- The benchmark does not remove hallucinations, but it makes downstream behavior inspectable: a system that lands on the right answer may still have accepted the false premise for the wrong reasons.
Key claims
PHRBench classified whether models complied with, avoided or heuristically corrected a false premise across 4,820 controlled cases.
Qualification: Hallucinated context can become the premise for another model's reasoning, so final-answer accuracy alone cannot show how the error was handled.
Evidence: source-2026-10-08-014
Sources
- arXiv preprint 2610.10455arXiv · primary research
Corrections
No corrections have been recorded for this story.