What is the Reconstruction benchmark?

It’s a test that gives an AI model only a scientific paper’s pre-publication reference list and asks it to guess the paper’s actual idea, while an independent AI judge checks whether the guess matches.

Did any single model do well on its own?

No. Individual match rates ranged from about 3% to 15%, with Claude Opus 4.8 scoring best at a 13.3% average.

Does combining multiple models help?

Yes. A multi-model cross-review-and-tournament pipeline lifted match rates to roughly 23-42%, averaging 36% — though that still means most ideas went unrecovered.

AI Models Recover Only 3–15% of Research Ideas, Study Finds

Q: Which AI models were tested?

Seven flagship models: Claude Opus 4.8, GPT-5.6 Sol Pro, Kimi K3, GLM 5.2, Gemini 3.1 Pro Preview, DeepSeek-V4-Pro, and Qwen3.7-Max.

A new benchmark suggests today’s most capable large language models still struggle to reconstruct a scientific paper’s core idea from only the list of prior work it cites — a stress test for claims that AI can meaningfully assist with scientific research.

Researchers behind the benchmark, called Reconstruction, withhold the seed paper along with any literature published at the same time or afterward, so a model cannot simply have read the study during training. The reference list is anonymized and frozen to what was citable before publication, then a model is asked to propose the paper’s actual contribution using only that reading list. An independent AI judge checks whether the guess matches the real idea.

The team ran the AI benchmark on 643 papers spanning six fields: machine learning, astronomy, chemistry, materials science, medicine, and physics.

Frontier models mostly missed it

Seven current flagship models were tested individually: Claude Opus 4.8, GPT-5.6 Sol Pro, Kimi K3, GLM 5.2, Gemini 3.1 Pro Preview, DeepSeek-V4-Pro, and Qwen3.7-Max. None fared well — match rates clustered between 3% and 15% depending on domain, with Claude Opus 4.8 posting the best individual average at 13.3%, according to the paper.

Combining models raised the score considerably. A pipeline that had multiple models cross-review each other’s guesses, then ran a tournament-style selection among the top four, reached a 22.9%-to-41.6% range across domains, averaging 36% — about 2.4 times better than the strongest single model.

Why it matters

The results arrive as AI labs increasingly market their models as research collaborators. OpenAI recently said its unreleased Astra model solved ten previously unsolved math problems, and Anthropic’s Claude Fable 5 helped disprove an 87-year-old math conjecture earlier this year. Reconstruction suggests a different, harder skill — inferring an idea nobody has stated yet from context alone — remains largely out of reach for a single model.

The authors describe Reconstruction as part of a larger “AI-Professor” project, with this test serving as a check on literature understanding before attempting a harder “Generation mode” aimed at proposing genuinely new ideas rather than recovering published ones. For now, the gap between recovering an existing idea and generating an original one looks wide even for the best multi-agent setups.

Frontier models mostly missed it

Why it matters

Frequently asked questions