safety security
Compiling Was a Bad Score for Vulnerability Repair
Harness artifacts and compiler flags moved the metric without proving a security fix.
Summary
Harness artifacts and compiler flags moved the metric without proving a security fix.
A study of 203 vulnerable C and C++ functions found compile rate to be an unreliable proxy for LLM vulnerability repair. About 64% of compile failures were not attributed to the model, and a compiler-standard flag changed compile rates by 1.8 to 2.7 times on identical patches. A compiler-feedback loop also rewarded deletions and placeholders while similarity to human fixes fell. The authors propose a change-aware screen as a cheap filter, not a substitute for execution-grounded security evaluation.
Why it matters
Harness artifacts and compiler flags moved the metric without proving a security fix.
Limits and context
- About 64% of compile failures were not attributed to the model, and a compiler-standard flag changed compile rates by 1.8 to 2.7 times on identical patches.
- The authors propose a change-aware screen as a cheap filter, not a substitute for execution-grounded security evaluation.
Key claims
Harness artifacts and compiler flags moved the metric without proving a security fix.
Qualification: About 64% of compile failures were not attributed to the model, and a compiler-standard flag changed compile rates by 1.8 to 2.7 times on identical patches.
Evidence: source-2026-09-23-004
Sources
- arXiv preprint 2609.26749arXiv · primary research
Corrections
No corrections have been recorded for this story.