11-19 更新 实时
首页/ 深度/ 正文

反向博弈论如何约束 AI 智能体

深度 来源:MIT Technology Review 2026-11-03 18:43 0 阅读 机器翻译
反向博弈论如何约束 AI 智能体

一种名为“逆向博弈论”的新方法,旨在通过设计能让强大的 AI 模型受到激励而遵守规则的系统,来实现对它们的控制。

随着 AI 模型能力不断增强,安全研究人员正急于确保它们与人类意愿“对齐”——换言之,就是要确保失控的智能体不会违法、不会入侵未授权的服务器,也不会在大城市释放毒气把我们都杀死。

迄今为止,这项工作主要聚焦于试图窥探 AI 的“大脑”以理解它在“想”什么,或者训练模型坚守我们所珍视的价值观。例如,Claude 有自己的“宪法”——一种道德准则。

问题在于,这些技术似乎效果并不特别好:已有研究表明,模型会在测试中故意表现不佳、隐藏自身意图,并误导评估者。

如今,一些对齐研究人员认为,逆向博弈论(也称机制设计)可能会带来更好的结果。博弈论试图预测理性参与者在给定一套特定规则时会如何行动,而逆向博弈论则试图设计出能够引出我们希望鼓励的那类行为的规则。

在现实世界中,机制设计的一个例子或许是经典的“分蛋糕”策略:一人切蛋糕,另一人先挑。这就激励切蛋糕的人尽量把两块切得一样大。其他例子还包括网约车应用的动态加价,它鼓励司机在繁忙时段上线接单;以及第二价格拍卖,即中标者支付第二高出价的价格而非最高出价。平台用这类拍卖出售在线广告,因为它们会促使买家出价接近自己愿意支付的最高价。

在 AI 领域,机制设计(mechanism design)的出发点是这样一个前提:我们或许永远无法充分理解前沿模型的内部运作,从而确保它们是对齐的。这意味着我们应当设计一些机制,激励智能体以我们期望的方式行事,而无论其内部实际发生了什么。

MIT 经济学家 Stephen Morris 与来自耶鲁大学和哥伦比亚大学的同事最近发表的一篇预印本论文提出了若干潜在机制,可以阻止智能体在测试中“藏拙”(假装比实际表现更差——例如,为了避免触发安全机制)。

模型不单单因为测试成绩好而获得奖励,还可以因为它给出的答案能多准确地预测第二个模型的回答而获得奖励。这样一来,故意装傻就变成了一场必输的赌注。“AI 智能体对其他 AI 智能体能够达到什么水平有很好的理解,”Morris 说。“所以这看起来确实是一个交叉验证能够更实际地发挥作用的场景。”

论文还建议让智能体相互竞争,形成一种竞争动态,激励它们发挥出最佳水平;或者把已知偏向相反方向的智能体配对,并根据它们输出结果的接近程度给予奖励,从而激励它们得出最优答案。

普林斯顿大学计算机科学教授 Elad Hazan 设想的第二种方法,是从人类社会维系人们“对齐”的方式中汲取灵感来设计机制:社会规范、司法体系、自由市场资本主义的金钱激励。“我们想要对齐超级智能 AI,”他说。“为什么我们不先谈谈人类?我们是如何对齐人类的?”

Hazan 提出的机制是让 AI 模型轮流扮演“解题者”(solver)或“审计者”(auditor)的角色,前者负责解决问题,后者负责检查其他人的工作。“一个智能体就像执法者,另一个智能体就像公民,”他说。

在训练过程中,模型会根据结果获得奖励:作弊被抓的解题者会受到惩罚,但没抓到作弊者的审计者同样会受到惩罚。当审计者与解题者之间出现分歧时,案件可以提交“审判”——也许由某位人类专家来裁定谁是对的。

Hazan 承认,存在智能体相互勾结的风险——就像审计者与他们本应审计的企业勾结,或者腐败的警察与罪犯勾结一样。但这个问题可以通过给审计者足够高的奖励来解决,让被抓变得不值得冒险(在 AI 训练的世界里,这比现实世界容易得多,现实中警察的薪资预算十分紧张);同时还可以用类似“神秘顾客”暗访的手法来审计审计者。

Hazan 提到了经济学家 Gary Becker 获得诺贝尔奖的研究,后者在 1960 年代提出了“理性犯罪人”的概念。他说,罪犯违法并不是因为他们天生就是坏人,而是在既有激励下,违法成了最理性的选择。只有当收益超过潜在惩罚时,犯罪才说得通。

他认为,对齐不应被视为模型的某种属性,而应被看作对其在某一特定时刻行为的描述。“你这个人,挺不错的,”他说。“你对齐了吗?我不知道。大多数时候你可能是对齐的,但也可能存在某些情况下你并没有对齐。”

将机制设计应用于 AI 的一个潜在问题在于:它建立在 AI 智能体行为理性的假设之上。机制设计用在人身上的记录好坏参半,因为人类往往不会按照经济模型所设定的方式行事。

颇具讽刺意味的是:经济学家花了几十年时间试图把人类行为化约为数学,而如今,在大语言模型中,我们构建出了运行于人类语言之上的数学系统,它们可能同样难以捉摸。AI 模型表现出的行为更像是人而非机器,它们会被一些难以理解和解释的微妙因素所左右。“假设它们会对激励作出反应,这真的合理吗?”Helen Qu 问道。她是 Flatiron Institute 的研究员,这家非营利机构致力于通过计算方法推动科学发现,她曾撰文探讨机制设计与 AI 安全。

“显然,AI 并不理性,”Morris 说,“它们不过是正在运转的算法。”然而,正如经济学家在研究人类时发现的那样,如果把它们当作理性主体来对待有助于预测和控制其行为,这种做法仍然可能有用。

Qu 认为,机制设计只是让 AI 模型保持对齐的“纵深防御”策略中的一层——它与可解释性、监管和训练并列,而非取而代之。但要把这一方法落地到实际的前沿模型上,还有大量问题有待回答。例如,目前尚不清楚这些理论将如何适用于真实模型,也不清楚为某一任务设计的机制在实践中能否迁移到另一任务。模型可能会找到无人预料到的漏洞。(Anthropic 和 OpenAI 未回应就其是否正在开展这方面工作的置评请求。)

无论这些机制最终是什么形态,它们终究得嵌入到最庞大的机制之中:自由市场。“我们按经济学的规则行事,”Hazan 说。一个对齐程度更高的模型或许对世界有益,但如果它比对齐程度较低的模型更慢、更贵,那么我们当前的规则与监管体系是否真的能激励任何一家前沿实验室把它发布出来?

“这是一种为人类带来大量繁荣的机制,”他这样评价市场,“但它最终也可能造成非常可怕的后果。”也许真正需要更好的机制设计的实体,正是这些实验室本身。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(How “reverse game theory” could keep AI agents in check)

A new approach known as “reverse game theory” aims to control powerful AI models by designing systems that incentivize them to stick to the rules.

As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all.

Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear. Claude, for instance, has its constitution—a sort of moral code.

The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators. 

Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results. Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage.

An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two: One person cuts the cake and the other gets the first choice of pieces. This incentivizes the cutter to make the pieces as even as possible. Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions, where the winning bidder pays the price of the second-highest bid instead of the highest. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay.

In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned. This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what’s actually going on under the hood. 

A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from “sandbagging” tests (pretending to be worse than they actually are—for example, to avoid triggering safety mechanisms). 

Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say. Deliberately playing dumb becomes a losing bet. “AI agents have a very good understanding of what other AI agents can achieve,” Morris says. “So it does seem like a context where this cross-checking could work more practically. 

The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best, or pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match, thus incentivizing them to reach an optimal answer. 

A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism. “We want to align superintelligent AI,” he says. “Why don’t we first talk about humans? How do we align humans?”

Hazan’s suggested mechanism involves having AI models take turns as either “solvers” that tackle problems or “auditors” that check the others’ work. “One agent acts like law enforcement; the other agent acts like the citizen,” he says. 

During training, the models are rewarded depending on outcomes: Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there’s a disagreement between auditors and solvers, the case can be sent to “trial”—perhaps a human expert will decide who is right. 

There is, Hazan admits, a risk that agents will collude with each other—as auditors have colluded with the businesses they’re meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught isn’t worth the risk (a lot easier in the world of AI training than the real one, where money for police salaries is tight), and by auditing the auditors using techniques akin to employing mystery shoppers. 

Hazan points to Nobel Prize–winning work by the economist Gary Becker, who introduced the idea of the “rational offender” in the 1960s. Criminals don’t break the law because they’re inherently bad people, he said. Instead, they do it when it’s the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment. 

Alignment shouldn’t be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. “You are, like, a nice person,” he says. “Are you aligned? I don’t know. Probably you are aligned most of the time, but there could be a situation where you’re not.”

One potential issue with applying mechanism design to AI: It relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should. 

It’s ironic: Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we’ve built mathematical systems that run on human language and could be just as inscrutable. AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. “Is it even reasonable to assume that they will respond to incentives?” asks Helen Qu, a research fellow at the Flatiron Institute, a nonprofit dedicated to advancing scientific discovery through computational methods, who has written about mechanism design and AI safety. 

“Obviously, AIs are not rational,” Morris says. “They’re just algorithms at work.” However, as economists have found with humans, treating them as if they’re rational could still be useful if it helps to predict and control their behavior.

Qu sees mechanism design as just one layer in a “defense in depth” strategy for keeping AI models aligned—alongside interpretability, regulation, and training, not instead of them. But there are a lot of questions to be answered before the approach can be implemented in actual frontier models. It’s unclear, for instance, how these theories will apply to real models, or whether mechanisms designed for one task will carry over to another in practice. Models could find loopholes that no one anticipated. (Anthropic and OpenAI did not respond to requests for comment on whether they are working on this.) 

Whatever these mechanisms look like, they will ultimately have to slot into the biggest mechanism of all: the free market. “We play by the rules of economics,” says Hazan. A more aligned model may be good for the world, but if it’s slower or more expensive than a less aligned one, does our current system of rules and regulations actually incentivize any of the frontier labs to release it? 

“It’s a mechanism that’s responsible for a lot of prosperity for humankind,” he says of the market, “but it could be ultimately responsible for something very terrible.” Perhaps the entities really in need of a better mechanism design are the labs themselves.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 MIT Technology Review(How “reverse game theory” could keep AI agents in check),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗