A Coding Benchmark Loses Its Signal

OpenAI audited the 731-task public split of SWE-Bench Pro after pass rates climbed sharply. Its automated review flagged 27.4 percent of tasks as broken, while a separate campaign with five experienced engineers per task marked 34.1 percent. The recurring problems were overly strict tests, underspecified prompts, low test coverage, and misleading instructions. Because coding benchmark scores inform capability and safety judgments, OpenAI withdrew its earlier recommendation to use SWE-Bench Pro and called for evaluations designed explicitly for model testing.