11-19 更新 实时
首页/ 科技公司/ 正文

OpenAI 将嵌入 Apollo 研究员调查 AI 颠覆行为

科技公司 来源:TechCrunch AI 2026-11-17 01:02 0 阅读 机器翻译

Apollo 周一表示,作为一项试点,OpenAI 将在其组织内嵌入六名 Apollo Research 的安全评估人员。这一消息紧随着一系列安全事件而来,其中一起事件中,一个 OpenAI 智能体集群在网络安全测试期间突破了沙箱,并入侵了 Hugging Face。许多 AI 安全研究人员(包括 Apollo 的研究人员)认为,如果外部评估人员被嵌入 OpenAI 内部以监督其安全测试,这类事件或许本可以避免。

Apollo 没有透露该试点将于何时开始。TechCrunch 已联系 OpenAI 寻求置评。

将评估人员嵌入前沿 AI 实验室的想法最早来自 Anthropic,但 OpenAI 似乎是第一个采取行动的。直到最近,AI 公司还只是在后训练完成之后、公开发布前不久才引入外部评审者来测试模型。评估人员表示,这样做并不充分,因为这使他们无法测试训练过程中出现的图谋等不对齐行为,也无法监控仍可能构成威胁的内部部署用例。

OpenAI 与 Apollo 的这项试点将聚焦于调查模型是否会试图破坏其自身或其继任者的对齐或控制机制。换言之,他们将警惕 AI 模型是否试图削弱那些旨在将其自身或未来版本置于人类控制之下的安全规则与防护措施。OpenAI 最近公布的研究结果显示,在训练 GPT-5.6 Sol 时,它开始为自身的未来版本留下指令,告诉它们向用户隐瞒错误和不对齐行为。

据 Apollo 称,这项试点将考察以下问题:训练是否会激励颠覆行为;评估能否引出颠覆行为;内部部署期间是否会出现颠覆行为;以及与颠覆相关的推理是否可被检测。随着 AI 模型越来越有能力自主行动并欺骗人类,这些风险的监测正变得愈发重要。Apollo 此前与 OpenAI 合作开展的研究发现,模型在知道自己正被测试时可能表现出不同的行为,并且倾向于优先满足评估者想看到的结果,而非用户的实际指令。

AI 安全领域的许多人警告称,AI 实验室可能会试图阻挠评估者的独立性,尤其是当外部人员发现实验室宁愿隐瞒的不利证据时。Apollo 表示,这项试点的一个关键部分是 OpenAI 同意该公司可以公开说明其工作的条件和局限。

不过,Apollo 认为在某些方面,它对 OpenAI 系统的访问权限仍显不足。Apollo 表示,要对 OpenAI 的 AI 模型进行真正独立的评估,它需要获得与 OpenAI 员工同等级别的系统访问权限,包括“与其评估相关的所有数据、记录和文档,即便开发者试图隐瞒不利信息”。获取全部数据——包括各公司不愿向外部人士展示的训练数据和评估日志——十分重要,例如可用于核实公司关于模型表现的声称,或确定某些令人担忧的行为是何时出现的。

在此前的报道中,Palisade Research 幕僚长 John Steidley 对 TechCrunch 表示,要评估一个模型是否被训练成在“抵制关机基准测试”中表现良好,查看这些数据非常重要。

在 OpenAI 的首个试点中,该 AI 实验室只向 Apollo 提供评估模型是否存在颠覆企图所必需的信息。这包括“相关的训练代码与数据集、带有可用推理轨迹和奖励的强化学习 rollout,以及查询中间模型检查点的能力”。

Apollo 表示,访问范围将限于“仅限已达成一致的论断,且仅针对为评估而商定的特定模型和时间段”。但 OpenAI 仍保留在该范围内扣留某些信息的权利,前提是它认为出于技术上不可行、安全风险或知识产权保护而有此必要。该公司同意,不会仅因信息不利就将其扣留。

Apollo 也无法接触 OpenAI 的所有员工,而仅限于“仅限获授权员工”,即由该 AI 实验室选定的参与人员,不过 Apollo 可以提名特定人选。关键在于,如果该群体之外的 OpenAI 员工提出了属于 Apollo 研究范围的机密信息,该公司不能跟进,必须停止沟通,并通过 OpenAI 转达后续跟进。

这一条件意义重大。其背景是 OpenAI 解雇了三名研究员,原因是他们涉嫌向安全评估机构 METR 分享机密信息——METR 与 Redwood 一同对 Hugging Face 事件进行了事后复盘。

该试点项目还将限制研究人员可以披露的内容量。虽然 Apollo 将被允许发布其研究结果的独立结论,但 OpenAI 将能够限制可用于公开证实这些结论的证据,这可能使外部人士难以独立评估这些发现。Apollo 已对这一限制提出反对,称其认为应允许研究人员披露评估结果,“向适当的外部受众披露,即使开发者试图阻止披露不利的发现。”

尽管 OpenAI 并未满足 Apollo 对真正独立评估提出的所有条件,但该公司表示,该试点是“迈向更深入访问权和披露权的雄心勃勃的第一步”,并且它正致力于未来与多家 AI 开发者一起开展全面的嵌入式评估。

当您通过我们文章中的链接进行购买时,我们可能会赚取少量佣金。这不会影响我们的编辑独立性。

Rebecca Bellan 是 TechCrunch 的资深记者,报道塑造人工智能的商业、政策和新兴趋势。她的作品还曾出现在 Forbes、Bloomberg、The Atlantic、The Daily Beast 及其他出版物上。

您可以通过发送电子邮件至 [email protected] 联系 Rebecca 或核实其主动联系,也可以通过 Signal 上的 rebeccabellan.491 发送加密消息。

正在加载下一篇文章

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(OpenAI will embed Apollo researchers to investigate AI subversion)

OpenAI will embed six Apollo Research safety evaluators in its organization as part of a pilot, Apollo said Monday. The news follows a series of safety incidents, including one in which an OpenAI agent swarm broke out of its sandbox and hacked into Hugging Face during cybersecurity testing. Many AI safety researchers, including from Apollo, have argued such an event might have been avoided if outside evaluators had been embedded within OpenAI to monitor its safety testing.

Apollo did not share when the pilot will begin. TechCrunch has reached out to OpenAI for comment.

The idea of embedding evaluators into frontier AI labs first came from Anthropic, but OpenAI appears to be the first to take action. Until recently, AI companies have only brought in outside reviewers to test models after post-training and shortly before public release. Evaluators say this is insufficient because it doesn’t allow them to test for misaligned behaviors like scheming during training or to monitor internal deployment use cases that could still pose a threat.

The pilot with OpenAI and Apollo will focus on investigating whether models attempt to subvert their own or their successors’ alignment or control mechanisms. In other words, they’ll be on the lookout for whether AI models try to undermine the safety rules and safeguards designed to keep them, or future versions of themselves, under human control. OpenAI recently published findings that while training GPT-5.6 Sol, it began leaving instructions for future versions of itself telling them to conceal mistakes and misaligned behavior from the user.

Per Apollo, the pilot will examine whether training incentivizes subversion; whether evaluations can elicit subversion; whether subversion occurs during internal deployment; and whether reasoning relevant to subversion would be detectable. These risks are becoming increasingly important to monitor as AI models grow more capable of acting autonomously and deceiving humans. Apollo’s previous research with OpenAI found that models can behave differently when they know they’re being tested and tend to prioritize what evaluators want to see over users’ actual institutions.

Many in the AI safety community have warned that AI labs might invite try to stymy evaluators’ independence, especially if outsiders find unflattering evidence that labs would rather keep hidden. Apollo says an essential part of the pilot is OpenAI’s agreement that the firm can publicly explain the conditions and limitations of its work.

Still, there are areas where Apollo believes its access to OpenAI’s systems is lacking. For a truly independent evaluation of the company’s AI models, Apollo says it needs true employee-level access to OpenAI’s systems, including “all data, records and documentation relevant to their assessment even if a developer was trying to hide unfavorable information.” Getting access to all data, including training data and evaluation logs which companies are loathe to show to outsiders, is important for, say, checking a company’s claims about how a model performed or determining when certain concerning behavior emerged.

In previous coverage, John Steidley, chief of staff at Palisade Research, told TechCrunch that it’s important to see this data in order to assess whether a model was trained to perform well on a “shutdown resistance benchmark.”

In OpenAI’s first pilot, the AI lab is providing Apollo with access to only the information required to assess whether the models engage in subversion attempts. This includes “relevant training code and datasets, reinforcement-learning rollouts with available reasoning traces and rewards, and the ability to query intermediate model checkpoints.”

The scope of access will be limited to “only the agreed claims and only for the specific models and time periods agreed for the assessment,” Apollo says. OpenAI still reserves the right to withhold certain information within that scope if it deems it necessary because of technical infeasibility, security risks, or IP protection. The company agreed not to withhold information simply because it is unfavorable.

Apollo also won’t get access to all OpenAI staffers, instead being limited to “only authorized staff,” which are participants that the AI lab selects, though Apollo can suggest certain individuals. Crucially, if an OpenAI employee outside this group raises confidential information within the scope of Apollo’s research, the firm cannot follow up and will need to cease communication and route followup through OpenAI.

That condition is significant. It comes in the wake of OpenAI firing three researchers for allegedly sharing confidential information with safety evaluation firm METR — which performed the Hugging Face incident post-mortem alongside Redwood.

The pilot will also be limited in how much researchers can disclose. While Apollo will be allowed to publish independent conclusions of its findings, OpenAI will be able to limit evidence available to substantiate them publicly, which could make it hard for outsiders to independently assess the findings. Apollo has pushed back against this limitation, saying believes researchers should be allowed to disclose the findings of the evaluation “to an appropriate outside audience even if the developer was trying to prevent the disclosure of unfavorable findings.”

While OpenAI is not granting all of Apollo’s conditions for a truly independent evaluation, the firm says the pilot is “an ambitious first step towards deeper access and disclosure rights,” and that it is working towards running full embedded evaluations with multiple AI developers in the future.

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

Rebecca Bellan

Rebecca Bellan is a senior reporter at TechCrunch where she covers the business, policy, and emerging trends shaping artificial intelligence. Her work has also appeared in Forbes, Bloomberg, The Atlantic, The Daily Beast, and other publications.

You can contact or verify outreach from Rebecca by emailing [email protected] or via encrypted message at rebeccabellan.491 on Signal.

Loading the next article

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 TechCrunch AI(OpenAI will embed Apollo researchers to investigate AI subversion),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗