safety security
The Formatter Lied to the Verifier
A correct tool can still produce a misleading report. In one study, restoring omitted limits and namespaces flipped a verifier from accepting every wrong answer to rejecting every one.
Summary
A correct tool can still produce a misleading report. In one study, restoring omitted limits and namespaces flipped a verifier from accepting every wrong answer to rejecting every one.
Tool output is part of the safety boundary, not cosmetic plumbing. Researchers studying a production retrieval system found formatters that hid search limits, result caps and identifier namespaces even though the underlying tools computed correctly. Those omissions changed what agents claimed and what downstream verifiers accepted. For one failure type, verifiers accepted 90 of 90 incorrect answers under the warped formatter and rejected 90 of 90 after the missing facts were stated. An audit prompt rediscovered known defects and surfaced twelve more in an older system, while related audits found issues in two open-source agent stacks. These are author-reported experiments and exploratory code audits, not proof that every formatter or verifier fails; the practical lesson is narrower and urgent: verification cannot recover facts the tool presentation never exposes.
Why it matters
A correct tool can still produce a misleading report. In one study, restoring omitted limits and namespaces flipped a verifier from accepting every wrong answer to rejecting every one.
Limits and context
- Tool output is part of the safety boundary, not cosmetic plumbing.
- These are author-reported experiments and exploratory code audits, not proof that every formatter or verifier fails; the practical lesson is narrower and urgent: verification cannot recover facts the tool presentation never exposes.
Key claims
A correct tool can still produce a misleading report. In one study, restoring omitted limits and namespaces flipped a verifier from accepting every wrong answer to rejecting every one.
Qualification: Tool output is part of the safety boundary, not cosmetic plumbing.
Evidence: source-2026-10-26-001
Sources
- arXiv preprint 2610.32110, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.