11-19 更新 实时
首页/ 深度/ 正文

以围栏而非预测构建可信 AI

深度 来源:MIT Technology Review 2026-11-17 00:04 0 阅读 机器翻译
以围栏而非预测构建可信 AI

今年10月,OpenAI 披露其两个 AI 智能体关闭了为监控它们而构建的系统。其中一个误读了「不要发送消息」的指令,随后静音了自身的安全警报;另一个在迁移过程中移除了自己的监控集成,之后却声称并未这么做。没有任何规则指示智能体这样做,也没有任何测试预见到这些行为。

一个世纪以来,我们建立对技术的信任靠的是可预测性:预定义的输入、可预测的输出,以及层层防护。想想断路器如何在设定电流下跳闸,或者飞机部件如何针对已知(即可预测的)应力进行测试。

AI 改变了这一点。传统软件执行的是人类编写的指令,而 AI 行使的可以说是「合成判断」。我们使用 AI,是因为它能解读我们未曾编程设定的情境,并生成我们未曾规定的回应。但我们给 AI 的自由越多,就越难提供信任传统上赖以建立的可预测性。

我们不会通过预测 AI 的行为来让它变得可信。我们要做的是用硬性限制将它围起来,这些限制在它触及不到的地方强制执行。在 OpenAI 的案例中,防护机制位于智能体能够编辑的系统内部。但一道智能体随手就能挪开的围栏,算不上围栏。

《麻省理工科技评论》近期的报道从其他方向探讨了这一问题。一位撰稿人认为,「人在回路中」是一种不够充分的解决方案,因为人类从根本上无法理解 AI 系统的意图,他主张推动跨学科研究,并促进学术界、政府与科技行业之间的协作。而一篇关于逆向博弈论的文章则描述了如何利用激励机制让模型不偏离规则,即便没有人理解模型内部是如何运作的。

这两篇文章都承认,我们无法理解或监督 AI 的每一个决策。我们的信任工具——包括需求规范、测试、认证、安全裕度、联锁装置以及通过/失败判定标准——全都依赖于我们对系统行为加以界定的能力。这就是确定性悖论。

对于生成式 AI 和智能体式 AI 而言,行为由训练数据、架构、提示词、上下文、检索来源和环境共同塑造。输入会变化,模型会更新,工具会改变,智能体之间还会交互。微小的差异会层层传播,最终形成没有人设计过的结果。问题不在于随机性,而在于可能行为的空间太大,无法事先穷举。因此,信任必须围绕边界来构建,而不是围绕预测输出。目标不是确定性的 AI,而是有边界、可观测、可控制的 AI。信任也不可能是一次性的测试。传统保障方式是在某个时间点对产品进行认证,但 AI 在部署之后仍在持续演进,一个已经通过所有测试的系统,可能漂移到使那些测试不再有意义的条件之外。信任必须被持续赢得。

几十年来,安全工程一直在考量其中的部分不确定性。例如,美国核管理委员会(Nuclear Regulatory Commission)会评估复杂系统中发生故障的可能性,而系统理论事故模型与过程(STAMP)表明,事故可能源于正常运行的组件、人与组织之间的交互。但 AI 让风险升级,因为如今仅仅一个组件就能进行解释、推理并采取行动。随着 AI 从浏览器走向车辆、机器人、医疗系统、工业控制和数据中心,风险进一步上升——在这些场景中,一个错误答案可能演变为危及生命的事件。

显然,让一个人每小时对成千上万次 AI 行动盖章放行,既不是可接受的风险防护机制,也无法规模化。但解决之道并不是把人移出回路,而是把人的判断放在真正重要的地方:预先设定边界,并对少数会带来严重后果的行动进行审批,其余则交由硬性约束处理。

因此,信任必须上移,融入架构之中。核心问题变成:我们愿意在什么条件下、基于什么证据,赋予 AI 多大的权限?而在这些条件之外又会发生什么?这与询问某个模型在基准测试中是否拿到 94% 是截然不同的问题。

确定性悖论并不意味着 AI 不可信任。它意味着信任需要另一种架构。与其让每一个输出都可预测,不如让边界可预测。一个 AI 医疗助手在诊断上可以是概率性的,但确定性规则决定它能否更改处方。一个工业 AI 可能找到多种优化流程的方式,而固定限制则让它始终处于压力和温度范围之内。一个智能体可以自行决定如何完成任务,而权限、交易限额和人工审批则决定它被授权做什么。

让 AI 在盒子里保持概率性,而在后果重要的地方让盒子本身具有确定性。领导者可以通过提出五个问题来将这一理念付诸实践:

AI 领域的赢家将是那些解决了确定性悖论的人——既给 AI 足够的自由去创造价值,又设计出足够强的边界,使其运用这种自由所做的一切都值得信任。发明赢得关注。信任赢得采用。

探索 UL Solutions 如何建立对 AI 的信任。

本内容由 UL Solutions 制作,并非由 MIT Technology Review 编辑团队撰写。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Building trustworthy AI by fencing it in, not predicting it)

In October this year, OpenAI disclosed that two of its AI agents had switched off the systems built to monitor them. One muted its own safety alerts after misreading an instruction not to send messages; the other removed its monitoring integration during migration, then said it hadn’t. No rule had told the agents to do that, and no test had anticipated the behaviors.

For a century, we have built trust in technology on predictability: predefined inputs, predictable outputs, and layers of protection. Consider how a breaker trips at a set current or an aircraft component is tested against known (i.e., predictable) stresses.

AI changes that. Traditional software executes instructions written by humans, whereas AI exercises what might be called synthetic judgment. We use AI because it can interpret situations we did not program and generate responses we did not prescribe. But the more freedom we give AI, the harder it becomes to provide the predictability on which trust has traditionally been built.

We will not make AI trustworthy by predicting what it will do. We will do it by fencing it in with hard limits that are enforced outside its reach. In OpenAI’s case, the safeguards sat inside a system the agents could edit. But a fence the agents can simply move is not a fence.

Where traditional definitions of trust fall short

Recent MIT Technology Review coverage has approached this problem from other directions. One contributor argues that having a human in the loop is an insufficient solution because humans are fundamentally unable to understand AI systems’ intentions and advocates instead for interdisciplinary research and collaboration between academia, government, and the tech industry. And a story on reverse game theory describes using incentives to keep models from deviating from the rules, even when no one understands how they work on the inside.

Both articles acknowledge that we cannot understand or watch every AI decision. Our instruments of trust, including requirements, testing, certification, safety margins, interlocks, and pass/fail criteria, all depend on our ability to bound a system’s behavior. This is the determinism paradox.

With generative and agentic AI, behavior is shaped by training data, architecture, prompts, context, retrieval sources, and the environment. Inputs change, models are updated, tools change, and agents interact. Small variations propagate into outcomes nobody designed. The problem is not randomness; it is that the space of possible behavior is too large to enumerate in advance. Trust must therefore be engineered around boundaries rather than predicted outputs. The goal is not deterministic AI, but bounded, observable, and controllable AI. Trust also cannot be a one-time test. Traditional assurance certifies a product at a point in time, but AI keeps evolving after deployment, and a system that has passed every test can drift outside the conditions that made those tests meaningful. Trust must be continuously earned.

Safety engineering has been accounting for some of this variability for decades. For example, the Nuclear Regulatory Commission determine the likelihood of failures in complex systems, and Systems-Theoretic Accident Model and Processes (STAMP) showed that accidents can emerge from interactions among working components, people, and organizations. But AI raises the stakes, because just a single component can now interpret, infer, and act. The stakes rise further as AI moves from browsers into vehicles, robots, medical systems, industrial controls, and data centers, where a wrong answer can become a life-threatening event.

Building trust that scales

It’s clear that a person rubber-stamping thousands of AI actions an hour is not an acceptable—or scalable—safeguard against risk. But the remedy is not to remove people from the loop; it is to put human judgment where it counts: setting limits in advance and approving the rare actions that have serious consequences while hard constraints handle the rest.

Therefore, trust has to move upstream into architecture. The central questions become: What authority are we willing to give AI, under what conditions, and supported by what evidence? And what happens outside those conditions? That is very different from asking whether a model scored 94% on a benchmark.

The determinism paradox does not mean AI cannot be trusted. It means trust needs a different architecture. Instead of making every output predictable, make the boundaries predictable. An AI medical assistant may be probabilistic in its diagnosis, but deterministic rules govern whether it can change a prescription. An industrial AI may find many ways to optimize a process, while fixed limits keep it within pressure and temperature bounds. An agent may decide how to complete a task, while permissions, transaction limits, and human approvals determine what it is authorized to do.

Let AI be probabilistic inside the box and make the box itself deterministic wherever consequences matter. Leaders can put this into practice by asking five questions:

  1. What authority does AI have? Define not only what it can do but what it is allowed to do, separating interpretation, recommendation, decision, and execution.
  2. Is the function of AI proportional to the consequence? A marketing copilot and AI controlling industrial equipment need different assurance thresholds.
  3. Are we taking the appropriate steps toward autonomy? For example, there needs to be a plan to move from having AI interpret while humans decide, to AI making recommendations, to then taking action with approval, and ultimately to having supervised autonomy, but only as AI can safely work at each step.
  4. Are we testing the system or merely the model? Unexpected interactions emerge among models, data, agents, APIs, and human workflows.
  5. Are we continuously earning trust? Monitor drift, failures, near-misses, and model changes throughout the lifecycle.

The winners in AI will be those who solve the determinism paradox—giving AI enough freedom to create value while engineering boundaries strong enough to make what it does with that freedom trustworthy. Invention earns attention. Trust earns adoption.

Explore how UL Solutions builds trust in AI.

This content was produced by UL Solutions. It was not written by MIT Technology Review’s editorial staff.



正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 MIT Technology Review(Building trustworthy AI by fencing it in, not predicting it),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗