benchmarks evals
A Successful Final Answer Can Hide an Impossible Path
Prefix executability checks every partial tool-call sequence against schemas, state preconditions and data dependencies before terminal success can conceal the failure.
Summary
Prefix executability checks every partial tool-call sequence against schemas, state preconditions and data dependencies before terminal success can conceal the failure.
Tool-use benchmarks usually grade the end state, even when an intermediate call could never have executed. Prefix executability measures whether every partial trajectory remains valid under tool schemas, state preconditions and data dependencies. Across BFCL-v3, ToolSandbox and the Airline and Retail splits of tau3-bench, the authors found that outcome-only and prefix-level rankings diverged: plan-conditioned agents sometimes improved final success while reducing executable-trajectory survival. The proposed curves, first-error depth and recovery measures expose where invalidity begins and whether later success follows recovery. Explicit plan checking improved prefix survival while keeping terminal performance competitive. The protocol diagnoses benchmark traces; it does not by itself prevent bad tool calls in deployment.
Why it matters
Prefix executability checks every partial tool-call sequence against schemas, state preconditions and data dependencies before terminal success can conceal the failure.
Limits and context
- Across BFCL-v3, ToolSandbox and the Airline and Retail splits of tau3-bench, the authors found that outcome-only and prefix-level rankings diverged: plan-conditioned agents sometimes improved final success while reducing executable-trajectory survival.
- The protocol diagnoses benchmark traces; it does not by itself prevent bad tool calls in deployment.
Key claims
Prefix executability checks every partial tool-call sequence against schemas, state preconditions and data dependencies before terminal success can conceal the failure.
Qualification: Across BFCL-v3, ToolSandbox and the Airline and Retail splits of tau3-bench, the authors found that outcome-only and prefix-level rankings diverged: plan-conditioned agents sometimes improved final success while reducing executable-trajectory survival.
Evidence: source-2026-10-30-006
Sources
- arXiv preprint 2610.40444, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.