benchmarks evals
A Coding Benchmark Loses Its Signal
An OpenAI audit estimates that roughly thirty percent of SWE-Bench Pro's public tasks are broken or unfairly specified.
Summary
An OpenAI audit estimates that roughly thirty percent of SWE-Bench Pro's public tasks are broken or unfairly specified.
OpenAI audited the 731-task public split of SWE-Bench Pro after pass rates climbed sharply. Its automated review flagged 27.4 percent of tasks as broken, while a separate campaign with five experienced engineers per task marked 34.1 percent. The recurring problems were overly strict tests, underspecified prompts, low test coverage, and misleading instructions. Because coding benchmark scores inform capability and safety judgments, OpenAI withdrew its earlier recommendation to use SWE-Bench Pro and called for evaluations designed explicitly for model testing.
Why it matters
An OpenAI audit estimates that roughly thirty percent of SWE-Bench Pro's public tasks are broken or unfairly specified.
Limits and context
No additional limitation was separately recorded.
Key claims
An OpenAI audit estimates that roughly thirty percent of SWE-Bench Pro's public tasks are broken or unfairly specified.
Evidence: source-2026-07-10-012
Sources
- OpenAI: Separating signal from noise in coding evaluationsopenai.com · secondary reporting
Corrections
No corrections have been recorded for this story.