OpenAI has retracted its recommendation that developers rely on SWE-Bench Pro, a widely used benchmark for evaluating how well AI models handle real-world coding tasks, after an internal audit found that close to a third of the benchmark’s problems are flawed, the company said this week.
What the audit found
SWE-Bench Pro tests models on coding tasks pulled from GitHub issues and pull requests, checking whether a proposed fix passes a set of tests without breaking other functionality. OpenAI built an automated review pipeline that flagged 200 of the benchmark’s 731 tasks, or 27.4%, as broken. A follow-up human review found problems in 249 tasks, or 34.1%.
According to OpenAI, the errors fall into a few recurring patterns: tests that check for unspecified implementation details, task prompts that omit requirements a model has no way to infer, test suites with such low coverage that incomplete fixes still pass, and prompts that steer solutions toward the wrong approach entirely.
Why the benchmark breaks down
OpenAI says the root problem is where the tasks come from. SWE-Bench Pro’s problems are drawn from real GitHub issues, written by maintainers for human contributors working through an extended back-and-forth, not designed as clean, self-contained evaluations. Tests written to check one specific patch during a code review often end up stricter than the original task description ever asked for.
The company also noted that reported model performance on the benchmark has climbed from a 23.3% pass rate to 80.3% in eight months, a pace it says is approaching a noise ceiling of around 70% — making it hard to tell whether models are truly improving or simply exploiting quirks in the test set.
What OpenAI recommends instead
Rather than patch SWE-Bench Pro after the fact, OpenAI is urging the field to build new coding benchmarks with experienced software developers and stronger human oversight from the start, and to treat existing SWE-Bench Pro scores with skepticism in the meantime.