benchmarks evals
A Correct Answer Could Still Carry an Invalid Trace
Trace Integrity separates reference-answer accuracy from executable, schema-valid and replayable computation.
Summary
Trace Integrity separates reference-answer accuracy from executable, schema-valid and replayable computation.
On BIRD Mini-Dev, three SQL-agent variants posted answer accuracies of 20, 22 and 24 percent, while their trace-integrity pass rates were 39, 43 and 40 percent. The authors’ Correct Answer / Invalid Trace rates remained 55, 59.1 and 45.8 percent, showing that answer matching, trace validity and silent-failure risk measured different things in this demonstration.
Why it matters
Trace Integrity separates reference-answer accuracy from executable, schema-valid and replayable computation.
Limits and context
No additional limitation was separately recorded.
Key claims
Trace Integrity separates reference-answer accuracy from executable, schema-valid and replayable computation.
Evidence: source-2026-08-27-003
Sources
- arXiv preprint 2608.26036arXiv · primary research
Corrections
No corrections have been recorded for this story.