OpenAI Finds 30% of SWE-Bench Pro Tasks Broken, Retracting Earlier Recommendation
Updated
Updated · OpenAI · Jul 8
OpenAI Finds 30% of SWE-Bench Pro Tasks Broken, Retracting Earlier Recommendation
3 articles · Updated · OpenAI · Jul 8
Summary
OpenAI said about 30% of SWE-Bench Pro’s 731 public coding tasks are broken, undermining a benchmark it had previously urged developers to adopt.
A screening pipeline flagged 286 suspect tasks, then investigator agents and five software engineers per task reviewed them and found flaws such as overly strict tests, underspecified prompts, low-coverage tests and misleading instructions.
The company said those defects can both fail correct solutions and pass incomplete ones, distorting model capability measurements that feed into deployment and safety decisions.
Frontier models’ pass rates had climbed from 23.3% to 80.3% in eight months on the benchmark, but OpenAI now warns developers to scrutinize those results carefully.
The audit follows OpenAI’s earlier criticism of SWE-bench Verified and broadens its argument that coding benchmarks built from repository history need stronger human oversight and validation.
If top AI coding benchmarks are fundamentally broken, how much of the recent progress in AI is actually an illusion?
With AI now smart enough to outwit its own tests, what does a truly un-gameable evaluation system even look like?
Benchmark Crisis: DeepSWE Uncovers High Error Rates and Exploits in SWE-Bench Pro
Overview
In May 2026, the AI community introduced DeepSWE, a new benchmark designed to set a higher standard for evaluating AI coding agents. DeepSWE was created in response to the flaws of earlier benchmarks, aiming to give a clearer and more practical view of how advanced models perform. It features 113 original, hand-written tasks across 91 projects, with no public GitHub history, which prevents data contamination and ensures models cannot simply recall solutions. This approach helps reveal real performance differences between top coding agents, making DeepSWE a more reliable and rigorous tool for measuring AI progress.