OpenAI Retracts Endorsement of SWE-Bench Pro Coding Benchmark
An internal audit found that roughly a third of SWE-Bench Pro's coding tasks are flawed, prompting OpenAI to withdraw its recommendation that developers rely on the benchmark.
Read more →AI Benchmarks
AI benchmarks are standardized tests used to compare how capable different AI models are on tasks like reasoning, coding, and math. This hub covers what the major benchmarks measure, why the scores don’t always tell the full story, and the news when new results land. Expect explainers that help you read the numbers critically.
Combine with:
Grok 4.5 Launches Cheaper Than Opus, With Mixed Benchmarks
SpaceXAI released Grok 4.5, its first model trained on Cursor data, priced well below Anthropic's Opus 4.8 — though its own self-reported benchmarks show mixed results against it.
Read more →China's GLM-5.2 Rivals US Frontier AI at One-Sixth the Cost
Z.ai has released GLM-5.2, a 744-billion-parameter open-weight model that matches GPT-5.5 on key benchmarks and trails Claude Opus 4.8 narrowly — trained entirely on Huawei silicon with no Nvidia hardware.
Read more →Meta AI Chief Claims In-Training "Watermelon" Model Has Caught Up to GPT-5.5
Meta's superintelligence chief told employees that the company's next model, codenamed Watermelon, has caught up to OpenAI's GPT-5.5 on key benchmarks — though the specific tests and results have not been disclosed.
Read more →Mistral Releases Leanstral 1.5 to Automate Math Proofs and Code Verification
Mistral's new open-source model for Lean 4 fully saturates the miniF2F theorem benchmark, solves 87% of graduate-level math tests, and found five previously unknown bugs in production code — at roughly $4 per proof versus $300 for competing systems.
Read more →What Is an AI Benchmark — and How Do You Know If a Model Is Actually Better?
An AI benchmark is a standardized test used to compare how capable different AI models are. Here's what the major ones measure — and why the scores don't always tell the full story.
Read more →OpenAI Releases GeneBench-Pro to Test AI in Biology Research
OpenAI has released GeneBench-Pro, a 129-problem benchmark that tests whether AI agents can exercise the judgment-intensive analytical reasoning required for real-world computational biology research.
Read more →Z.ai's GLM-5.2 Matches US Frontier AI at One-Sixth the API Cost
GLM-5.2, released on June 16 by Beijing startup Z.ai, sits alongside Claude Opus 4.8 and GPT-5.5 on coding benchmarks while carrying an API price roughly one-sixth that of closed US models.
Read more →ByteDance Doubao Seed 2.1 Claims Parity With GPT-5.5 on Agent Tasks
ByteDance launched Doubao Seed 2.1 Pro and Turbo on June 23, claiming top scores on coding and agent benchmarks and parity with GPT-5.5 and Claude Opus 4.7.
Read more →Zhipu AI's GLM-5.2 Beats GPT-5.5 on Coding at a Sixth of the Price
Chinese lab Zhipu AI has open-sourced GLM-5.2, a 753-billion-parameter model that outperforms GPT-5.5 on software engineering benchmarks while costing roughly one-sixth as much.
Read more →Alibaba Open-Sources AI World Model That Tops GPT-5.4 on Agent Tasks
Alibaba's Qwen team has released Qwen-AgentWorld, a family of open-source language world models that predict how digital environments respond to AI agent actions, with its 397B variant outscoring GPT-5.4 on the new AgentWorldBench benchmark.
Read more →