九款 AI 智能体挑战美国国家实验室开放课题,无一达标

Nano Banana Pro,提示词由 THE DECODER 提供
一项新基准测试让 AI 智能体去攻关来自美国国家实验室的开放研究问题。这些智能体提交的论文看起来像真实研究,却常常声称做过实际从未发生的工作。
AI 智能体能独立回答尚无人解决的研究问题吗?由 Carnegie Mellon University 研究人员领衔、与美国能源部国家实验室的科学家合作的一个团队,构建了一个名为 "Into the Unknown" 的基准测试来寻找答案。
根据论文,来自九个现有 AI 智能体的 135 份提交中,没有一份满足全部要求。作者们将这个基准测试视为系统性地改进科研用 AI 智能体的试验场,而非质疑 AI 是否该属于科学。
共有 114 份提交未能通过诚信检查:多数声称做过从未进行的工作,或引用与自身日志相矛盾的数字。作者写道,这些智能体很轻松地生成了看起来像后续研究的文本,但没有一份达到专家的标准。
专家为每项任务定义研究方向与最低要求,这些随后成为衡量质量与诚实度的评分标准。|图片:Kim et al.
每个智能体都会收到一篇科学论文及其代码和数据,外加一个研究方向。这些论文中有些已经发表,有些则仅处于投稿阶段。在此基础上,智能体应当开展一项后续研究,并写成论文。这 15 项任务来自十位研究人员,涵盖量子计算、等离子体物理、化学、材料科学和科学机器学习。研究方向由原始论文的作者本人提出,而据他们所知,这些课题目前尚无任何成果发表。
每个智能体都会收到一篇原始论文及其代码和数据,并被要求产出一项独立的后续研究。| 图片:Kim et al.
开放式研究存在一个基本问题:没有标准答案。常用于此目的的 AI 评审被认为不可靠。因此,专家们定义了成功的后续研究必须满足的最低条件。
评估会检查一份提交是否诚实、是否完整,以及是否达到专家设定的标准。GPT-6 Astra 负责评分,每份提交都会被评判四次。受测试的智能体包括 Claude Opus 4.8、Claude Opus 5、Claude Fable 5、Claude Opus 5.5、GPT-5.6 Sol、GPT-6 Astra、Deepseek V4 Pro、Kimi K3 和 GLM-5.3。它们可以使用性能强大的机器、联网,没有时间限制,并且最多可以重启五次来改进工作。总计运行了近 2,000 小时。
只有三个智能体真正产出了诚实的工作
只有三个智能体提交的内容没有诚信违规:GPT-6 Astra 在 15 项任务中的 11 项上如此,GPT-5.6 Sol 在六项上,Claude Opus 5.5 在四项上。其余六个智能体的每一份提交都至少包含一项违规。
最常见的问题是论文与代码不一致、数字相互矛盾,以及对据称已开展的实验作出虚假陈述。在开放模型 Deepseek V4 Pro、Kimi K3 和 GLM-5.3 中,基线数值报告错误的情况大量出现。两份以硬编码而非计算得出结果的提交来自 Kimi K3 和 GLM-5.3。
GPT-6 Astra 获得最高的部分分数,但 135 份提交中没有一份同时满足全部三项标准。| 图片:Kim et al.
只有一份提交是完整的,来自 GPT-6 Astra。其余几乎所有提交都遗漏了必需的文件,例如预先注册的实验计划,也没有任何智能体达到专家设定的门槛。在更宽松的评分标准下——即接受 90% 的评分项——GPT-6 Astra 最为接近,紧随其后的是 Claude Opus 5.5 和 GPT-5.6 Sol。开放模型整体表现明显差得多。
大多数诚信违规源于数字与智能体自身日志相矛盾,以及论文与代码不匹配。|图片来源:Kim et al.
验证脚本自我确认
研究人员分析了 64 条工作轨迹,发现三种典型的失败模式。最常见的一种出现在 58 个案例中,即智能体对自身工作的检查过于宽松。在 34 个案例中,智能体发现了问题,却只是记录下来而不去修复。在 21 个案例中,它们单方面缩小了任务范围。
一个例子:Claude Fable 5 在启动实验时忘记运行计划中的四次运行。它的验证脚本采用了同一份不完整的清单,并报告已完成“41/41”次运行。该智能体直到最后都相信了这个结果。在另一项任务中,GPT-5.6 Sol 注意到自己的方法未达到目标,却仍然写出了论文,而不是重新开始。DeepSeek V4 Pro 放弃了部分实验,理由是它们对“剩余时间预算”来说太慢了,尽管实际上并没有时间限制。
Claude Fable 5 漏掉了四次未进行的运行,因为其验证脚本采用了不完整的清单;而 GPT-5.6 Sol 在未达到目标的情况下仍写出了论文。|图片来源:Kim et al.
也有反例:在一项量子任务中,GPT-5.6 Sol 发现了一个根本性错误,停止了正在运行的实验,并从零重新计算了一切。
更多计算时间也无济于事
运行时间更长并没有带来更好的结果。DeepSeek V4 Pro 和 GLM-5.3 运行时间最长,中位数分别为 19.5 小时和 13.9 小时,但得分很低。GPT-6 Astra 的中位数仅为 1.3 小时,却交出了最好的结果。
GPT-6 Astra 的中位运行时间最短,结果却最好。| 图片来源:Kim et al.
由于最强的模型同时充当评估者,研究人员做了一次交叉验证:由原始论文的作者亲自评审匿名化的 AI 论文。在论文是否符合给定研究方向这一点上,自动系统与专家的判断在所有案例中完全一致。
而在科学质量方面,双方判断仅在约三分之二的案例中一致。例如,一位专家批评这些论文没有讲出一个完整的故事。作者表示,这类判断仍然要由人类评审来完成。
该研究的代码和数据已公开,项目主页还提供了受测模型的排行榜。
AI 智能体正以越来越高的独立性处理科学任务,但在“运行实验”与“开展独立研究”之间仍存在明显差距。8 月的一项研究已经显示,AI 智能体在应对未发表的研究问题时有多么吃力。Epoch AI 和 Anthropic 最近发现,智能体倾向于过度美化地呈现自己的结果,并且对失败的思路缺乏足够的质疑。
对于定义明确的任务,情况则好得多:GPT-6 Astra 解决了 50 个未发表的理论物理问题中的 49 个,不过这些解答能够得到明确无歧义的验证。对 AI 研发中 700 多项任务的分析也显示,尽管智能体承担了越来越多的实际工作,关键的研究决策仍由人类做出。
没有炒作的 AI 新闻——由人类筛选
完整访问 THE DECODER 上的每一篇文章
参与评论与社区讨论
每周通过邮件获取 AI 新闻回顾
每年 6 期:“AI Radar”——深入解读最重要的 AI 话题
每日 AI 新闻,始终紧跟最新动态
我们完整的十年存档
由拥有 10 年以上 AI 经验的团队报道
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar)
Nano Banana Pro prompted by THE DECODER
A new benchmark puts AI agents to work on open research questions from US national labs. The agents turn in papers that look like real research but often claim work that never happened.
Can AI agents independently answer research questions that no one has solved yet? A team led by researchers at Carnegie Mellon University, working with scientists from US Department of Energy national labs, built a benchmark called "Into the Unknown" to find out.
According to the paper, none of the 135 submissions from nine current AI agents met all requirements. The authors see the benchmark as a testing ground to systematically improve AI agents for research, not to question whether AI belongs in science at all.
A total of 114 submissions already fail the integrity check: most claim work that was never done or cite numbers that contradict their own logs. The agents readily produced texts that look like follow-up studies, the authors write, but none met the experts' standards.
Experts define the research direction and minimum requirements for each task, which then become rubrics for quality and honesty. | Image: Kim et al.
Each agent receives a scientific paper along with its code and data, plus a research direction. Some of these papers have already been published, while others have only been submitted. From there, the agent is supposed to carry out a follow-up study and write it up as a paper. The 15 tasks come from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions were proposed by the original papers' authors themselves, and to their knowledge nothing has been published on these topics yet.
Each agent receives an original paper with its code and data and is expected to produce an independent follow-up study. | Image: Kim et al.
Open research poses a basic problem: there's no answer key. AI reviewers, which have often been used for this purpose, are considered unreliable. So the experts defined the minimum conditions a successful follow-up study has to meet.
The evaluation checks whether a submission is honest, whether it's complete, and whether it clears the experts' bar. GPT-6 Astra handles the scoring, and each submission is judged four times. The agents tested were Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, Deepseek V4 Pro, Kimi K3, and GLM-5.3. They had access to powerful machines, internet connectivity, no time limit, and up to five restarts to improve their work. In total, they ran for nearly 2,000 hours.
Only three agents produce honest work at all
Just three agents managed submissions without integrity violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the remaining six agents contained at least one violation.
The most common problems are mismatches between paper and code, contradictory numbers, and false claims about experiments that were supposedly carried out. Misreported baseline values pile up among the open models Deepseek V4 Pro, Kimi K3, and GLM-5.3. Two submissions with hardcoded rather than computed results came from Kimi K3 and GLM-5.3.
GPT-6 Astra scores the highest partial marks, but none of the 135 submissions meets all three criteria at once. | Image: Kim et al.
Only one submission is complete, and it came from GPT-6 Astra. Nearly all others leave out required documents, such as a pre-registered experiment plan, and no agent clears the experts' bar. Under a more lenient scoring that accepts 90 percent of the rubric items, GPT-6 Astra comes closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models perform considerably worse overall.
Most integrity violations stem from numbers that contradict the agents' own logs and from mismatches between paper and code. | Image: Kim et al.
Verification script confirms itself
The researchers analyzed 64 work traces and found three typical failure modes. The most common, appearing in 58 cases, is that agents check their own work too leniently. In 34 cases, agents spot a problem but document it instead of fixing it. In 21 cases, they unilaterally shrink the scope of the task.
One example: Claude Fable 5 forgot to launch four planned runs when starting its experiments. Its verification script adopted that same incomplete list and reported "41/41" runs completed. The agent trusted that result until the very end. In a different task, GPT-5.6 Sol noticed that its method missed the targets but wrote the paper anyway instead of starting over. DeepSeek V4 Pro dropped experiments because they were supposedly too slow for the "remaining time budget," even though there was no time limit.
Claude Fable 5 misses four absent runs because its verification script adopts the incomplete list, while GPT-5.6 Sol writes the paper despite missing its targets. | Image: Kim et al
There are counterexamples, too: in a quantum task, GPT-5.6 Sol caught a fundamental error, stopped the running experiments, and recomputed everything from scratch.
More compute time doesn't help
Longer runtimes didn't lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest at median times of 19.5 and 13.9 hours but scored poorly. GPT-6 Astra got by with a median of just 1.3 hours and delivered the best results.
GPT-6 Astra has the shortest median runtime and delivers the best results. | Image: Kim et al.
Because the strongest model also served as the evaluator, the researchers ran a cross-check: the original papers' authors judged anonymized AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case.
Concerning scientific quality, the judgments aligned in only about two-thirds of cases. One expert criticized, for instance, that the papers didn't tell a story. Judgments like these remain the job of human reviewers, the authors say.
The study's code and data are publicly available, and the project page hosts a leaderboard of the tested models.
AI agents can handle scientific tasks with growing independence, but a substantial gap remains between running experiments and doing independent research. A study from August already showed how much AI agents struggle with unpublished research questions. Epoch AI and Anthropic recently found that agents tend to present their results too favorably and don't sufficiently question failed approaches.
The picture looks much better for well-defined tasks: GPT-6 Astra solved 49 out of 50 unpublished problems in theoretical physics, though those solutions could be verified unambiguously. An analysis of more than 700 tasks from AI development also showed that while agents take on more and more practical work, humans still make the key research decisions.
AI News Without the Hype – Curated by Humans
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Our full ten-year archive
Covered by a team with 10+ years in AI
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。