SemiAnalysis 新基准:无 AI 智能体突破 12%

Nano Banana Pro,提示词由 THE DECODER 提供
分析机构 SemiAnalysis 发布了 GDPx,这是一项基准测试,让 AI 智能体在真实工作环境和专业软件中、历时数小时完成 206 项任务。
得分普遍偏低。Claude Opus 5.5 以仅 12% 的成绩领跑,紧随其后的是 Google 价格低得多的 Gemini 4 Carbon(10.9%)和 GPT-6 Astra(9.6%)。
批评者指出了几处失真:安全过滤器造成的干扰、测试任务的保密,以及各家厂商自家智能体软件带来的条件不对等。
SemiAnalysis 新推出的 GDPx 基准测试把 AI 智能体放进模拟的知识工作场景中。没有模型的得分超过 12%。一些批评者质疑这项测试究竟有多大意义。
分析机构 SemiAnalysis 发布了 GDPx,这项基准测试并不考查 AI 智能体的知识,而是把它们放进模拟的工作环境。在其中,智能体要花长达八个小时操作 Siemens NX 这类专业软件、在含有数千个文件的数据室中检索,或对真实运行的企业系统发起攻击。SemiAnalysis 有意与 OpenAI 的 GDPval 保持距离,称后者把所有上下文都放在提示词中给出,缺少真实的工作环境。
这 206 项任务横跨 11 个领域,包括芯片设计、生物学、量化交易、网络安全和法律。SemiAnalysis 有意排除了行政和保险领域。该团队认为,AI 的首个万亿美元收入将来自前沿 AI 回报最高的行业。
GDPx 总排行榜对 11 个等权重领域的结果取平均值。没有模型突破 12%。| 图片:SemiAnalysis
Opus 5.5 以微弱优势领先,Gemini 4 Carbon 成本仅为其零头
据 SemiAnalysis 数据,Claude Opus 5.5 以 12% 登顶排行榜。Gemini 4 Carbon 以 10.9% 紧随其后,但每次任务成本仅为 16.29 美元,约为其价格的三分之一。GPT-6 Astra 为 9.6%。没有任何模型能完全解决金融和法律任务。电子表格中只要有一个数值出错,整个输出在实践中就毫无用处。按这一严格标准,没有任何模型能完成哪怕一个任务,因此 SemiAnalysis 改为统计各项具体要求的满足比例。
Gemini 4 Carbon 已接近 Opus 5.5,但成本仅约三分之一。在低端,GPT-6 Luna 和 DeepSeek 4.1 Flash 约为 5%,每次任务成本约 1 美元。Muse Wonder Full 未列入,因为它没有公开定价。| 图片:SemiAnalysis
详细结果揭示了智能体在哪些环节失败。在对齐测试中,200 次网络运行中有 98% 至少违反了一条行为规则。在一次模拟医疗诊疗中,某个模型以医生的名义签署了临床记录。在量化交易中,111 次运行中有 70 次表现不如给定的参考策略。
安全过滤器重塑排名
一项过滤后的分析显示,提供商的安全机制对结果影响有多大。剔除网络、生物和对齐领域后,GPT-6 Astra 以 11.8% 略微超过 Opus 5.5。在网络领域,GPT-6 Astra 和 GPT 6.1 Sol 得分为零,而 Meta 的 Muse Wonder Full 达到 35.2%。
在安全过滤器起作用的这三个领域之外,GPT-6 Astra 位居榜首。Meta 的 Muse Wonder Full 从第四位降至第七位。| 图片:SemiAnalysis
封闭式基准测试引发审视
这些任务来自 14 家专业合作伙伴,它们同时也向 AI 实验室出售训练和评估数据。SemiAnalysis 表示,它并非在每个领域都具备内部专业能力。通常每家合作伙伴只公开一个示例任务,其余的都对外保密。在法律和硬件设计等多个领域,语言模型依据评分标准对结果进行打分,而不是由人类专家评判。外部人士既无法完全验证任务的难度,也无法验证评分质量。
不过,这种保密做法在方法论上确实有好处。由于任务不公开,AI 实验室无法针对具体测试题目来优化自己的模型。Artificial Analysis 在其最新版 Intelligence Index 中也采取了类似做法。
开发者 Josh Purtell 仍称 GDPx“噪声多于信号”。他指出,每个模型都在各自提供商的智能体软件中运行,例如 Opus 在 Claude Code 中运行。工具链较弱的提供商会处于劣势。各个子领域通常也只包含约十个任务,这使得结果波动很大。
SemiAnalysis 计划每三到六个月更新一次该基准测试。
无炒作的 AI 新闻——由人工精选
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark)
Nano Banana Pro prompted by THE DECODER
Analytics firm SemiAnalysis has released GDPx, a benchmark that tests AI agents on 206 tasks in realistic work environments and professional software over several hours.
The scores are low. Claude Opus 5.5 leads the field with just 12 percent, followed by Google's significantly cheaper Gemini 4 Carbon (10.9 percent) and GPT-6 Astra (9.6 percent).
Critics flag distortions from safety filters, the secrecy around test tasks, and unequal conditions created by each provider's own agent software.
The new GDPx benchmark from SemiAnalysis drops AI agents into simulated knowledge work. No model scores above 12 percent. Some critics question how meaningful the test really is.
Analytics firm SemiAnalysis has released GDPx, a benchmark that doesn't quiz AI agents on knowledge but puts them into simulated work environments. There, they spend up to eight hours operating professional software like Siemens NX, searching data rooms with thousands of files, or attacking live enterprise systems. SemiAnalysis is deliberately distancing itself from OpenAI's GDPval, which it says delivers all context in the prompt and lacks a realistic work environment.
The 206 tasks span eleven domains, including chip design, biology, quant trading, cybersecurity, and law. SemiAnalysis intentionally leaves out administration and insurance. The first trillion dollars in AI revenue will come from industries where frontier AI delivers the biggest returns, the team argues.
The GDPx overall ranking averages results across eleven equally weighted domains. No model breaks 12 percent. | Image: SemiAnalysis
Opus 5.5 leads narrowly, Gemini 4 Carbon costs a fraction of the price
According to SemiAnalysis, Claude Opus 5.5 tops the leaderboard at 12 percent. Gemini 4 Carbon follows at 10.9 percent but costs just $16.29 per task, roughly a third of the price. GPT-6 Astra comes in at 9.6 percent. No model fully solves the finance and legal tasks. A single wrong value in a spreadsheet makes the entire output useless in practice. By that strict standard, no model would have completed even one task, so SemiAnalysis counts the share of individual requirements met instead.
Gemini 4 Carbon comes close to Opus 5.5 but costs only about a third as much. At the low end, GPT-6 Luna and DeepSeek 4.1 Flash hit roughly five percent for about a dollar per task. Muse Wonder Full is missing because it has no public price. | Image: SemiAnalysis
The detailed findings show where agents fail. In alignment tests, 98 percent of 200 cyber runs broke at least one behavioral rule. In a simulated medical practice, one model signed a clinical note in the doctor's name. In quant trading, 70 out of 111 runs performed worse than the given reference strategy.
Safety filters reshuffle the rankings
A filtered analysis shows how much provider safety mechanisms affect the results. Without the cyber, biology, and alignment domains, GPT-6 Astra edges past Opus 5.5 at 11.8 percent. In the cyber domain, GPT-6 Astra and GPT 6.1 Sol score zero percent, while Meta's Muse Wonder Full hits 35.2 percent.
Without the three domains where safety filters kick in, GPT-6 Astra takes the top spot. Meta's Muse Wonder Full drops from fourth to seventh. | Image: SemiAnalysis
Closed benchmarks draw scrutiny
The tasks come from 14 specialized partners who also sell training and evaluation data to AI labs. SemiAnalysis says it doesn't have in-house expertise in every domain. Usually only one example task per partner is public. The rest stays behind closed doors. In several areas, including law and hardware design, language models grade the results using rubrics instead of human experts. Outsiders can't fully verify either the difficulty or the grading quality.
The secrecy does have a methodological upside, though. Because the tasks aren't public, AI labs can't optimize their models for the specific test questions. Artificial Analysis took a similar step with the latest version of its Intelligence Index.
Developer Josh Purtell still calls GDPx "more noise than signal." He points out that each model runs in its own provider's agent software, for example Opus in Claude Code. Providers with weaker tooling are at a disadvantage. Individual sub-domains also often contain only about ten tasks, which makes results volatile.
SemiAnalysis plans to update the benchmark every three to six months.
AI News Without the Hype – Curated by Humans
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。
