benchmarks evals
The Tool Benchmark Passed Keywords, Not Commands
A diagnostic ladder found that small-model tool-use claims could survive weak lexical checks while failing execution structure.
Summary
A diagnostic ladder found that small-model tool-use claims could survive weak lexical checks while failing execution structure.
The Keyword Harnesses Fail Open study examines evaluations that reward outputs for containing expected words rather than forming valid tool calls. Its staged diagnostic separates lexical overlap from argument binding, ordering and executable structure, exposing cases in which a reported success disappears under stricter checks. The lesson is narrow but practical: tool-use claims need verifiers tied to the real interface contract, not a permissive keyword proxy.
Why it matters
A diagnostic ladder found that small-model tool-use claims could survive weak lexical checks while failing execution structure.
Limits and context
- The lesson is narrow but practical: tool-use claims need verifiers tied to the real interface contract, not a permissive keyword proxy.
Key claims
A diagnostic ladder found that small-model tool-use claims could survive weak lexical checks while failing execution structure.
Qualification: The lesson is narrow but practical: tool-use claims need verifiers tied to the real interface contract, not a permissive keyword proxy.
Evidence: source-2026-10-03-008
Sources
- arXiv preprint 2610.02142arXiv · primary research
Corrections
No corrections have been recorded for this story.