safety security
A Good Answer Could Still Skip the Required Check
ContractEval maps query-active procedural obligations to trace evidence and separates omissions, wrong branches and invariant breaches.
Summary
ContractEval maps query-active procedural obligations to trace evidence and separates omissions, wrong branches and invariant breaches.
Output-only scoring can approve an answer even when the agent skipped the branch, check or dependency that justified it. ContractEval turns a procedure into obligations activated by the current query, then matches those obligations against a response or execution trace. Under gold expected and observed graphs, it localized every injected structural failure in a controlled audit suite; language-model extraction retained much of the signal but remained sensitive to calibration. The authors explicitly frame the method as an audit aid, not a compliance guarantee.
Why it matters
ContractEval maps query-active procedural obligations to trace evidence and separates omissions, wrong branches and invariant breaches.
Limits and context
- Output-only scoring can approve an answer even when the agent skipped the branch, check or dependency that justified it.
- The authors explicitly frame the method as an audit aid, not a compliance guarantee.
Key claims
ContractEval maps query-active procedural obligations to trace evidence and separates omissions, wrong branches and invariant breaches.
Qualification: Output-only scoring can approve an answer even when the agent skipped the branch, check or dependency that justified it.
Evidence: source-2026-09-10-007
Sources
- arXiv preprint 2609.09458arXiv · primary research
Corrections
No corrections have been recorded for this story.