developer tools
Passing Tests Covered Only a Third of the Required Behavior
On Req2Test-Bench, the strongest agent configuration produced runnable tests for 82.8 percent of cases but behaviorally adequate tests for 33.8 percent.
Summary
On Req2Test-Bench, the strongest agent configuration produced runnable tests for 82.8 percent of cases but behaviorally adequate tests for 33.8 percent.
A generated test can execute cleanly while failing to constrain the behavior that a requirement actually describes. Req2Test-Bench reconstructs 5,086 developer-authored behavior-driven-development instances from eight open-source projects, five languages and four frameworks. Agents receive a Gherkin requirement and target interfaces, then must generate the withheld test implementation. The best of eight evaluated configurations produced passing tests for 82.8 percent of instances, but only 33.8 percent killed every mutant killed by the developer test. Under-specified setup and implicit expected outcomes accounted for much of the information gap. The benchmark measures fidelity to recovered project artifacts; it does not establish that every developer-written test is complete.
Why it matters
On Req2Test-Bench, the strongest agent configuration produced runnable tests for 82.8 percent of cases but behaviorally adequate tests for 33.8 percent.
Limits and context
- The best of eight evaluated configurations produced passing tests for 82.8 percent of instances, but only 33.8 percent killed every mutant killed by the developer test.
- The benchmark measures fidelity to recovered project artifacts; it does not establish that every developer-written test is complete.
Key claims
On Req2Test-Bench, the strongest agent configuration produced runnable tests for 82.8 percent of cases but behaviorally adequate tests for 33.8 percent.
Qualification: The best of eight evaluated configurations produced passing tests for 82.8 percent of instances, but only 33.8 percent killed every mutant killed by the developer test.
Evidence: source-2026-10-28-006
Sources
- arXiv preprint 2610.36584, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.