实测:AI 编程智能体近四分之三任务仍需人工清理

ChatGPT-5.6,提示词由 THE DECODER 提供
来自阿里巴巴和几所亚洲高校的研究人员基于真实用户变更请求构建了一个编程智能体基准测试。即便是最好的模型也只能完整完成约四分之一的任务,这让此前那些接近满分的旧测试成绩显得耐人寻味。
该团队开发了 Co-Design Bench,这是一项针对编程智能体的新测试。智能体必须根据用户请求改造现有的 Web 应用,有时还需要以图像、音频录音、视频或文档作为输入。
根据论文,在受测的十个模型中表现最好的 GPT-6 Astra 平均能满足一项任务约 70% 的要求,但只有 26.4% 的任务能完全正确地完成。多数情况下,智能体会完成大部分工作,但至少留下一处细节未完成,因此使用这些工具的人几乎总要手动收尾。
编程基准测试对智能体而言已变得过于简单
作者以现有基准测试已趋饱和为依据,论证这项新测试的必要性。当模型能够自行运行、检查并修复代码时,它们在这些测试上的得分会超过 99%,模型之间的差异几乎无法衡量。
在 WebCompass-edit 等较早的基准测试上,智能体式编程模型取得了接近满分的成绩,而 Co-Design Bench 则留下了超过 35 个百分点的差距。| 图片:Wei 等人
Co-Design Bench 转而取材于真实编程会话中的实际请求。研究人员更倾向于选取用户不得不反复改写请求或回滚更改的案例,因此测试中主要包含那些在实践中已经给 AI 智能体造成困难的任务。这些任务既有小改动,比如从收据中提取一个金额并更新所有相关总计,也有整个项目级别的重构。
在多模态赛道中,智能体需要从收据、表格或视频中提取数值,并一致地更新所有依赖视图。| 图片来源:Wei et al.
评分标准同样严格。一名 AI 裁判在浏览器中测试完成的 App,随后第二名 AI 裁判复核每一个通过的判定,并查找是否存在原型图或硬编码结果。只有真正在 App 中可用的功能才能得分。
一个任务中包含多种变更类型会让模型不堪重负
排在 GPT-6 Astra 之后的是 Claude Opus 5 和 Qwen3.8-Max,分列第二、三位,而 Grok 4.6 垫底。在规模更大的改造任务中,体量小得多的 Qwen3.8-27B 升至第二位,但作者并未从这一结果中推断出该模型具有普遍优势。
当一项任务组合了更多类型的变更——包括布局、设计、交互和内容——所有模型的表现都会下降。但这些相互交织的需求在真实项目中才是常态
一旦任务要求三种或四种不同类型的变更,这十个模型的得分无一例外都会变差。| 图片来源:Wei et al.
增加算力对这种短板几乎无济于事。在较小的变更上,GLM-5.2 消耗的 Token 数量接近 GPT-6 Astra 的六倍,得分却仍明显更低。
GPT-6 Astra 在两个赛道均领先,但仅完全解决了 41% 的多模态任务,以及略低于 18% 的代码仓库任务。| 图片来源:Wei et al.
测试框架对结果的影响不亚于模型本身
在另一项测试中,研究人员让相同的模型运行在不同的框架(harness)中,也就是智能体用来读取、编写和执行代码的程序。Qwen3.8-27B 在较大规模改造任务上的得分,会因环境不同而是 75% 或仅为 56%。两个模型中哪一个更胜一筹,同样取决于环境。
因此,只列出模型的排行榜只能说明部分情况。作者建议始终将模型与工作环境一并报告。对于挑选编程智能体的公司来说,这意味着要把这两个部分作为一个组合来测试。
这个新基准测试确实存在局限。每项任务只评估一个固定请求;用户的追问或任务中途的纠正不在测试范围内。AI 评委也可能出错。
研究机构 METR 的一项研究已经表明,基准测试结果与实际可用性之间可能相差甚远。项目负责人对 AI 生成代码的实际接受率,比广泛使用的 SWE-bench Verified 的得分低约 24 个百分点。
测试环境中的缺陷也可能让智能体在并未真正解决问题的情况下通过任务。不过,工作环境的影响并非一目了然。最近一项关于编程智能体的研究发现,更精细的框架几乎带不来任何可测量的收益,尽管它们消耗的 Token 最多可达三倍。
没有炒作的 AI 新闻——由人工精选
可完整访问 THE DECODER 上的每一篇文章
参与评论和社区讨论
每周通过邮件获取 AI 新闻回顾
每年 6 期:“AI Radar”——深入解读最重要的 AI 话题
每日 AI 新闻,始终紧跟最新动态
我们完整的十年存档
由拥有 10 年以上 AI 经验的团队报道
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test)
ChatGPT-5.6 prompted by THE DECODER
Researchers from Alibaba and several Asian universities built a coding-agent benchmark from real user change requests. Even the best model fully completes only about one in four tasks, putting the near-perfect scores of older tests into perspective.
The team developed Co-Design Bench, a new test for coding agents. The agents must rework existing web apps based on user requests, sometimes using images, audio recordings, videos, or documents as input.
According to the paper, GPT-6 Astra, the best of the ten models tested, meets about 70 percent of a task's requirements on average. But the result is fully correct in only 26.4 percent of tasks. In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.
Coding benchmarks have grown too easy for agents
The authors justify the new test by pointing to the saturation of existing benchmarks. When models can run, check, and fix their own code, they hit scores above 99 percent on those tests, and differences between models can barely be measured.
On older benchmarks like WebCompass-edit, agentic coding models reach near-perfect scores, while Co-Design Bench leaves a gap of more than 35 percentage points. | Image: Wei et al.
Co-Design Bench draws on real requests from actual coding sessions instead. The researchers favored cases where users had to rephrase their request multiple times or roll back changes, so the test contains mainly tasks that already gave AI agents trouble in practice. They range from small tweaks, like pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls.
In the multimodal track, agents need to extract values from receipts, tables, or videos and update all dependent views consistently. | Image: Wei et al.
Grading is strict to match. One AI judge tests the finished app in a browser, then a second re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.
Multiple change types in one task overwhelm the models
Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max take the next spots, while Grok 4.6 lands in last place. On larger overhauls, the much smaller Qwen3.8-27B climbs to second, but the authors do not derive a general advantage of the model from that result.
Performance drops across all models as a task combines more types of changes, including layout, design, interaction, and content. But these intertwined requirements are the norm in real projects
Every one of the ten models scores worse once a task demands three or four different types of changes. | Image: Wei et al.
Extra compute does little to fix this weakness. GLM-5.2 burns through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse.
GPT-6 Astra leads both tracks but fully solves only 41 percent of multimodal tasks and just under 18 percent of repository tasks. | Image: Wei et al.
The harness shapes results as much as the model
In a separate test, the researchers ran the same models in different harnesses, the programs through which an agent reads, writes, and executes code. Qwen3.8-27B scored 75 or just 56 percent on larger overhauls depending on the environment. Which of two models came out ahead also depended on the environment.
Leaderboards that list only the model therefore only tell part of the story. The authors recommend always reporting model and working environment together. For companies picking a coding agent, that means testing both components as a pair.
The new benchmark does have limits. It evaluates one fixed request per task; follow-up questions or mid-task corrections from users are not part of the test. The AI judges can also make mistakes.
A study by the research organization METR already showed how far benchmark results and real-world usability can diverge. The actual acceptance rate of AI-generated code by project leads was roughly 24 percentage points below scores on the widely used SWE-bench Verified.
Flaws in test environments can also let agents pass tasks without actually solving them. The effect of the working environment is not clear-cut, though. A recent study on coding agents found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.
AI News Without the Hype – Curated by Humans
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Our full ten-year archive
Covered by a team with 10+ years in AI
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。