OpenAI Retracts Endorsement of SWE-Bench Pro Coding Benchmark
An internal audit found that roughly a third of SWE-Bench Pro's coding tasks are flawed, prompting OpenAI to withdraw its recommendation that developers rely on the benchmark.
Read more →AI Benchmarksand AI Tools
Every AI News story tagged with both AI Benchmarks and AI Tools — the two topics side by side, updated as new articles publish.
6 articles
Grok 4.5 Launches Cheaper Than Opus, With Mixed Benchmarks
SpaceXAI released Grok 4.5, its first model trained on Cursor data, priced well below Anthropic's Opus 4.8 — though its own self-reported benchmarks show mixed results against it.
Read more →Mistral Releases Leanstral 1.5 to Automate Math Proofs and Code Verification
Mistral's new open-source model for Lean 4 fully saturates the miniF2F theorem benchmark, solves 87% of graduate-level math tests, and found five previously unknown bugs in production code — at roughly $4 per proof versus $300 for competing systems.
Read more →What Is an AI Benchmark — and How Do You Know If a Model Is Actually Better?
An AI benchmark is a standardized test used to compare how capable different AI models are. Here's what the major ones measure — and why the scores don't always tell the full story.
Read more →ByteDance Doubao Seed 2.1 Claims Parity With GPT-5.5 on Agent Tasks
ByteDance launched Doubao Seed 2.1 Pro and Turbo on June 23, claiming top scores on coding and agent benchmarks and parity with GPT-5.5 and Claude Opus 4.7.
Read more →Zhipu AI's GLM-5.2 Beats GPT-5.5 on Coding at a Sixth of the Price
Chinese lab Zhipu AI has open-sourced GLM-5.2, a 753-billion-parameter model that outperforms GPT-5.5 on software engineering benchmarks while costing roughly one-sixth as much.
Read more →