safety security
The Repair Runs Cleanly and Returns the Wrong Answer
A preregistered study finds typical-input gates release 41 percent of wrong repairs after runtime or API migrations.
Summary
A preregistered study finds typical-input gates release 41 percent of wrong repairs after runtime or API migrations.
One edge input derived from the release note cuts that rate to 6.5 percent across 73 tasks. Hard distractors remain difficult, with 15.3 to 16.8 percent of wrong programs passing two stronger frozen checks.
Why it matters
A preregistered study finds typical-input gates release 41 percent of wrong repairs after runtime or API migrations.
Limits and context
- Hard distractors remain difficult, with 15.3 to 16.8 percent of wrong programs passing two stronger frozen checks.
Key claims
A preregistered study finds typical-input gates release 41 percent of wrong repairs after runtime or API migrations.
Qualification: Hard distractors remain difficult, with 15.3 to 16.8 percent of wrong programs passing two stronger frozen checks.
Evidence: source-2026-11-06-019
Sources
- arXiv preprint 2611.08269, original v1arXiv · primary research
Corrections
No corrections have been recorded for this story.