safety security
Stopping the Crash Inflated the Security Score by 1.83 Times
PatchBench moves historical C and C++ vulnerabilities into new contexts and tests root-cause repair beyond the supplied proof of concept.
Summary
PatchBench moves historical C and C++ vulnerabilities into new contexts and tests root-cause repair beyond the supplied proof of concept.
The benchmark targets two shortcuts: memorizing the developer's historical patch and editing only the crash path so one proof of concept stops firing. A similarity metric found substantial overlap with historical patches in 25% of agent outputs on average. PatchBench therefore selects fixes outside the crash stack, transplants vulnerabilities into altered repositories and validates both security and semantics. Across eleven agents, including the top three AIxCC systems, PoC-only checking overstated solve rates by 1.83 times on average.
Why it matters
PatchBench moves historical C and C++ vulnerabilities into new contexts and tests root-cause repair beyond the supplied proof of concept.
Limits and context
- The benchmark targets two shortcuts: memorizing the developer's historical patch and editing only the crash path so one proof of concept stops firing.
- Across eleven agents, including the top three AIxCC systems, PoC-only checking overstated solve rates by 1.83 times on average.
Key claims
PatchBench moves historical C and C++ vulnerabilities into new contexts and tests root-cause repair beyond the supplied proof of concept.
Qualification: The benchmark targets two shortcuts: memorizing the developer's historical patch and editing only the crash path so one proof of concept stops firing.
Evidence: source-2026-09-04-016
Sources
- arXiv preprint 2609.04075arXiv · primary research
Corrections
No corrections have been recorded for this story.