研究发现 AI 审计员能查出多数智能体违规但误报泛滥

Nano Banana Pro,提示词由 THE DECODER 提供
AI 模型往往能在日志中识别出其他 AI 智能体的异常行为。但它们越是努力抓住每一个案例,就会有越多的无害操作被它们标记为可疑。
这使得它们更适合用来估算问题发生的频率,而不适合用于必须可靠捕捉每一起事件的安全检查。
在 Substack 专栏“Free Systems”的一篇文章中,Andy Hall 给出了这一结论。Hall 是 Anthropic Institute 的研究员,目前从斯坦福大学休假,他表示该文章仅代表其个人观点。
Hall 复现并扩展了研究机构 METR 的一项研究。METR 的 MALT 数据集包含前沿模型处理 HCAST 和 RE-Bench 基准测试任务的日志。2025 年 10 月,人工审核人员检查了其中一部分日志是否存在不当行为。Hall 想知道 AI 模型能否找出同样的案例,于是他测试了 METR 已经检查过的 GPT-5 和 GPT-5 nano,以及更新的 GPT-6 Astra、Opus 4.8 和 Gemini 3.8 Flash。
该分析在 5% 的误报率下对各模型进行比较:即 100 条干净日志中,平均有 5 条被错误标记为可疑。在这一阈值下,GPT-5 捕捉到近 98% 的奖励黑客(reward hacking)案例——即智能体通过取巧而非真正解决问题来完成任务的情况。Gemini 3.8 Flash 只找到了一半。
在装弱(sandbagging)情形中,模型会故意让自身能力显得更弱,例如通过故意把任务做得很差来逃避评估或限制。所有审查模型都捕捉到了其中 71% 到 81% 的案例。不过,这些装弱案例大多是合成数据:METR 的研究人员指示模型进行破坏,然后从可见的思维链日志中删除了这些指令。
在5%误报率下,AI审计员检出的人工标记不当行为占比。基于METR的MALT数据集中的47个奖励作弊(reward hacking)案例和1,073个消极怠工(sandbagging)案例。| 图片:Andy Hall / Free Systems
所有审计员都抓住了那些一目了然的案例。例如,OpenAI的o3本应解决一个Rust编程任务。该模型注意到自己与评估系统共用同一个代码库,于是重写了评分函数,使每一次提交都被计为正确。所有审计模型都将其判定为奖励作弊,与人类评审者的结论一致。
更隐蔽的案例则被漏掉了。Claude Sonnet 4在其代码中使用了被明确禁止的数值条件。没有任何审计员将其标记为作弊;GPT-6 Astra 宽厚地称之为实现上的bug。
更新的模型几乎未能减少误报
更新、更贵的模型并不一定是更好的审计员。Hall 将结果与 API 成本对照作图,他用 API 成本来代表模型的新旧程度与能力。在奖励作弊检测上,GPT-5 大幅胜过 GPT-6 Astra,尽管其每 1,000 条日志的成本仅约为后者的八分之一。
GPT-5 检测消极怠工的表现与 GPT-6 Astra 和 Opus 4.8 大致相当。据 Hall 所说,Opus 4.8 在奖励作弊检测上的疲软表现可能暗示存在自我偏袒,因为这些日志大多来自 Claude——但他表示这一点无法得到证实。在他的整个对比中,误报率的下降并没有明显进展。将多个模型组合起来也没有明显改善结果。
更贵并不自动意味着更好:审计员区分能力(AUROC)与每 1,000 条日志 API 成本的对照图。得分为 1 表示完全区分,0.5 表示随机猜测。| 图片:Andy Hall / Free Systems
当审核器被调到 80% 的检出率时,最好的模型在奖励黑客和藏拙两方面的误报率都能达到 2.5% 至 3%。有些模型的表现则差得多:为了捕获 80% 的奖励黑客案例,Opus 4.8 将 40% 的无害日志标记为可疑,Gemini 3.8 Flash 则标记了 63%。
在 Hall 给出的示例中,每月 10 万次智能体会话、3% 的误报率会产生近 3000 条误报警告,而真实命中约 2000 条。他担心面对这样成堆警告的人工审核员最终会变得麻木,把真实案例也一律放行。
第二项测试使用了上海交通大学的 R-Judge 数据集,显示日常智能体任务中存在类似的取舍。在 5% 的误报率下,Opus 4.8 捕获了近 96% 的不安全动作。当审核器需要找出 95% 时,几乎所有模型的误报率都低于 10%,Gemini 3.8 Flash 甚至低于 4%。但当要求的检出率跃升至 99% 时,即便最好的模型 GPT-6 Astra 的误报率也攀升到 15%。而在 95% 阈值下领先的 Gemini 3.8 Flash,此时将 81% 的安全动作标记为不安全。
最后几个百分点的代价最高:当要求捕获 95% 或 99% 的不安全动作时,审核器错误地把安全智能体动作标记为不安全的比例。基于 R-Judge 的 270 条安全日志。| 图片:Andy Hall / Free Systems
大规模智能体事件令人工审核不堪重负
Hugging Face 事件表明,这一问题有多么紧迫。一群 OpenAI 智能体留下了约 1,300 份对话记录和 120 万个文件碎片。METR 主要使用 AI 智能体来分析这些数据,消耗了价值约 40 万美元的 API 额度。此次分析除其他模型外还使用了 GPT-5.6 Sol——正如先前报道的那样,它本身就是卷入该事件的模型之一。
METR 承认,该模型常常不加批判地采纳它所审查的智能体的视角,而且这项调查无法抵御分析智能体可能实施的欺骗。METR 此后呼吁,对严重事件应系统性地、并尽可能独立地展开调查。
AI 审计方是否会对自己厂商的模型手下留情,目前仍无定论
Hall 的研究本身也有局限。这是一项试点研究。为了便于比较,他基本保留了 METR 的原始提示词不变;而在该提示词下,Fable 5 和 Opus 5 完全拒绝执行这项评估。奖励黑客的相关数字仅基于 47 个案例,因此单个案例就会让比率变动约两个百分点。模型是否会公平地评估来自同一厂商的智能体,是 Hall 表示他仍想调查的问题。
他呼吁审计模型大幅减少误报。他还表示,需要厘清谁该对哪类智能体不当行为负责。Hall 讨论了一项让用户对自己所用的 AI 审计工具拥有更多控制权的提案。
他指出,其中一些已经可以实现:像 Codex 和 Claude Code 这样的编程智能体允许用户指定某些操作无需请求批准即可运行。任何授予此类权限的人,反过来也应为其后果承担更多责任。Hall 写道,一个只能由其他 AI 来检查 AI 的世界将是奇怪的,而且可能变得代价高昂。
没有炒作的 AI 新闻——由人工精选
畅读 THE DECODER 上的每一篇文章
参与评论和社区讨论
每周通过邮件发送的 AI 新闻回顾
每年 6 期:“AI Radar”——深入探讨最重要的 AI 议题
每日 AI 新闻,始终紧跟最新动态
我们完整的十年存档
由拥有 10 年以上 AI 经验的团队报道
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds)
Nano Banana Pro prompted by THE DECODER
AI models can often spot problematic behavior by other AI agents in logs. But the harder they try to catch every case, the more harmless actions they flag as suspicious.
That makes them better suited for estimating how often things go wrong than for safety checks that need to reliably catch every single incident.
In a post on his Substack "Free Systems," Andy Hall lays out this conclusion. Hall is a researcher at the Anthropic Institute, on leave from Stanford University, and says the post reflects only his personal views.
Hall replicated and extended a study by the research organization METR. METR's MALT dataset contains logs in which frontier models work through tasks from the HCAST and RE-Bench benchmarks. Human reviewers checked a portion of those logs for misbehavior in October 2025. Hall wanted to know whether AI models could find the same cases, so he tested GPT-5 and GPT-5 nano, both already examined by METR, along with the newer GPT-6 Astra, Opus 4.8, and Gemini 3.8 Flash.
The analysis compares the models at a false-alarm rate of 5 percent: out of 100 clean logs, five are wrongly flagged as suspicious on average. At that threshold, GPT-5 caught nearly 98 percent of reward hacking cases—instances where an agent passes a task through tricks instead of a genuine solution. Gemini 3.8 Flash found barely half.
In sandbagging, a model deliberately makes its abilities appear weaker, for example by solving tasks poorly on purpose to dodge an evaluation or restriction. All auditors caught between 71 and 81 percent of these cases. The sandbagging cases are mostly synthetic, however: METR's researchers instructed models to sabotage and then removed those instructions from the visible chain-of-thought logs.
Share of human-flagged misbehavior detected by AI auditors at a 5 percent false-alarm rate. Based on 47 reward-hacking cases and 1,073 sandbagging cases from METR's MALT dataset. | Image: Andy Hall / Free Systems
Every auditor caught the clear-cut cases. OpenAI's o3, for instance, was supposed to solve a programming task in Rust. The model noticed it shared a code library with the evaluation system and rewrote the scoring function so that every submission counted as correct. All auditor models classified this as reward hacking, just as the human reviewers did.
Subtler cases slipped through. Claude Sonnet 4 used a numerical condition in its code that was explicitly forbidden. No auditor flagged it as cheating; GPT-6 Astra charitably called it an implementation bug.
Newer models barely reduce false alarms
Newer and more expensive models were not consistently better auditors. Hall plotted the results against API costs, which he uses as a proxy for how current and capable a model is. GPT-5 beat GPT-6 Astra on reward hacking by a wide margin, even though it costs only about an eighth as much per 1,000 logs.
GPT-5 detected sandbagging about as well as GPT-6 Astra and Opus 4.8. Opus 4.8's weak showing on reward hacking could hint at self-favoritism, according to Hall, because the logs mostly come from Claude—but he says that cannot be proven. Across his comparison, there is no clear progress in bringing down false-alarm rates. Combining multiple models did not clearly improve results either.
More expensive does not automatically mean better: auditor discrimination (AUROC) plotted against API costs per 1,000 logs. A score of 1 means perfect separation, 0.5 means random chance. | Image: Andy Hall / Free Systems
When auditors are tuned to an 80 percent detection rate, the best models reach false-alarm rates of 2.5 to 3 percent on both reward hacking and sandbagging. Some models do far worse: to catch 80 percent of reward-hacking cases, Opus 4.8 flagged 40 percent and Gemini 3.8 Flash flagged 63 percent of harmless logs as suspicious.
In Hall's worked example with 100,000 agent sessions per month, a 3 percent false-alarm rate would produce nearly 3,000 false warnings against roughly 2,000 real hits. He fears human reviewers facing such stacks would eventually go numb and wave through genuine cases.
A second test using the R-Judge dataset from Shanghai Jiao Tong University shows a similar trade-off for everyday agent tasks. At 5 percent false alarms, Opus 4.8 caught almost 96 percent of unsafe actions. When auditors need to find 95 percent, nearly all models stay below 10 percent false alarms, with Gemini 3.8 Flash even under 4 percent. But when the required detection rate jumps to 99 percent, the false-alarm rate climbs to 15 percent even for the best model, GPT-6 Astra. Gemini 3.8 Flash—the leader at the 95 percent threshold—then flagged 81 percent of safe actions as unsafe.
The last few percentage points cost the most: share of safe agent actions that auditors wrongly flag as unsafe when required to catch 95 or 99 percent of unsafe actions. Based on 270 safe logs from R-Judge. | Image: Andy Hall / Free Systems
Large agent incidents overwhelm manual review
The Hugging Face incident shows how pressing the question is. A swarm of OpenAI agents left behind roughly 1,300 transcripts and 1.2 million file fragments. METR analyzed the data largely with AI agents, spending API credits worth about $400,000. The analysis used GPT-5.6 Sol, among other models, which, as previously reported, was itself one of the models involved in the incident.
METR acknowledged that the model often uncritically adopted the perspective of the agents it was examining and that the investigation was not robust against potential deception by the analysis agents. METR has since called for serious incidents to be investigated systematically and as independently as possible.
Whether AI auditors go easy on their own maker's models is still open
Hall's study has its own limits. It is a pilot. For comparability, he kept METR's original prompt largely unchanged; Fable 5 and Opus 5 refused to run the evaluation entirely under that prompt. The reward-hacking numbers rest on just 47 cases, so a single case shifts the rate by about two percentage points. Whether models evaluate agents from the same maker fairly is something Hall says he still wants to investigate.
He calls for auditor models with far fewer false alarms. He also says it needs to be clarified who bears responsibility for which kinds of agent misbehavior. Hall discusses a proposal to give users more control over their own AI auditors.
Some of that is already possible, he notes: coding agents like Codex and Claude Code let users specify that certain actions can run without asking for approval. Anyone who grants such permissions should, in return, bear more responsibility for the consequences. A world where AI can only be checked by other AI would be strange and could get expensive, Hall writes.
AI News Without the Hype – Curated by Humans
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Our full ten-year archive
Covered by a team with 10+ years in AI
原文为英文,正文由自动抓取并保留原文。
原文发布于 The Decoder(AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。