benchmarks evals
A Hundred Engineering Tickets Resist a Clean Sweep
ExpertCrowd-SWE uses sealed solutions, partial-credit graders and must-pass blockers across 87 unfamiliar open-source projects.
Summary
ExpertCrowd-SWE uses sealed solutions, partial-credit graders and must-pass blockers across 87 unfamiliar open-source projects.
ExpertCrowd-SWE contains 100 multi-file engineering tasks written by practicing engineers against pinned commits in 87 open-source projects and 11 languages. Each task has a sealed reference patch and a weighted grader with executable checks, partial credit and blockers for changes a maintainer would refuse. Across 3,495 scored runs, the leading evaluated agent averaged 67.3% and the next 63.7%, without a statistically significant gap. No task was solved fully by every model, and 80 tasks lacked a model that achieved full credit on all five runs.
Why it matters
ExpertCrowd-SWE uses sealed solutions, partial-credit graders and must-pass blockers across 87 unfamiliar open-source projects.
Limits and context
No additional limitation was separately recorded.
Key claims
ExpertCrowd-SWE uses sealed solutions, partial-credit graders and must-pass blockers across 87 unfamiliar open-source projects.
Evidence: source-2026-11-12-004
Sources
- arXiv:2611.13246v1arXiv / Cornell University · primary research
Corrections
No corrections have been recorded for this story.