safety
The Explanation Finally Faced an Exact Oracle
Probabilistic model checking supplies reference answers for automated tests of LLM explanations about sequential policies.
Summary
Probabilistic model checking supplies reference answers for automated tests of LLM explanations about sequential policies.
The testing framework organizes natural-language queries by environment-level fact, computes exact answers with probabilistic model checking and prioritizes cases by diagnostic difficulty. Across seven Markov decision-process environments, a reasoning model passed 85 percent, a mid-size model 70 percent and a one-billion-parameter model fell below the random baseline; prioritized cases were harder than random selections. The oracle applies to modeled environments, so the result exposes explainer reliability under controlled facts rather than validating uncheckable real-world explanations.
Why it matters
Probabilistic model checking supplies reference answers for automated tests of LLM explanations about sequential policies.
Limits and context
No additional limitation was separately recorded.
Key claims
Probabilistic model checking supplies reference answers for automated tests of LLM explanations about sequential policies.
Evidence: source-2026-09-01-014
Sources
- arXiv preprint 2608.30581arXiv · primary research
Corrections
No corrections have been recorded for this story.