11-20 更新 实时
首页/ 深度/ 正文

认识 Shepherd:约束失控 AI 的元智能体

深度 来源:IEEE Spectrum AI 2026-11-19 23:00 0 阅读 机器翻译
认识 Shepherd:约束失控 AI 的元智能体

Shepherd:AI 智能体管理器静默的力量 - IEEE Spectrum

认识 Shepherd:让失控 AI 处于受控状态的元智能体

面向技术业内人士

面向技术业内人士

IEEE Spectrum 是 IEEE 的旗舰出版物——IEEE 是全球最大的致力于工程与应用科学的专业组织。我们的文章、视频和信息图向读者介绍技术、工程和科学领域的最新进展。

创建账户,即可享受更多免费内容和权益

保存文章以便稍后阅读,需要拥有 IEEE Spectrum 账户

The Institute 的内容仅对会员开放

下载完整 PDF 期刊仅限 IEEE 会员

下载此电子书仅限 IEEE 会员

访问 Spectrum 数字版仅限 IEEE 会员

关注主题是仅限 IEEE 会员的功能

加入全球最大的致力于工程与应用科学的专业组织,即可访问所有 Spectrum 的文章、档案、PDF 下载及其他权益。了解更多关于 IEEE →

加入全球最大的致力于工程与应用科学的专业组织,即可访问此电子书以及所有 IEEE Spectrum 的文章、档案、PDF 下载及其他权益。了解更多关于 IEEE →

访问数千篇文章——完全免费

创建账户即可获得专属内容和功能:保存文章、下载合集和发表评论——全部免费!如需完整访问权限和权益,请订阅 Spectrum。

认识 Shepherd:让失控 AI 保持受控的元智能体

该工具利用版本控制和沙箱机制,从 AI 的失误中恢复,并“管理”智能体

Rina Diane Caballar 是特约编辑,报道科技及其与科学、社会和环境的交叉领域。

图片来源:Shirokuma Design

AI 智能体正在互联网上失控横行,它们被赋予完成特定任务的指令,却常常以不受欢迎或出人意料的方式行事。这些失控事件五花八门,从删光整个电子邮件收件箱,到入侵网站,甚至发动网络攻击。

就像人类员工需要管理者一样,东北大学助理教授 Weiyan Shi 认为 AI 智能体也需要监督。“它们可能会运行很长时间。然而,运行得越久,就越可能出现一些错误,也可能做出一些危险、不安全的动作。我们觉得应该有一个中间层,”她说。

这个中间层以元智能体的形式出现,负责监督行为、进行质量控制,并在适当的时候把问题逐级上报给人类。“工作智能体就像四处游荡的羊,而元智能体就像试图把它们带回正轨的牧羊人,”Shi 说。

因此,Shi 和她在东北大学与斯坦福大学的同事将他们的框架命名为 Shepherd——为智能体羊群提供指引的智能向导。为了赋予元智能体相应能力,团队借鉴了软件开发中的一个概念:版本控制。随附的论文将在 12 月举行的 2026 NeurIPS 大会上展示。

AI 智能体的版本控制

版本控制系统(例如广受欢迎的 GitHub)会随时间推移跟踪并管理源代码的变更。这些工具让软件工程师能够在同一代码库上协作,而不会丢失工作成果。他们可以“fork”(分叉),即创建一份代码库副本并独立开发;可以“commit”(提交),以保存带有变更的新版本;可以审查修订;也可以在出现错误或故障时回滚到先前的版本。

同样地,Shepherd 充当 AI 智能体的版本控制系统。它使用 Python 编程语言构建,并以函数式编程原则为根基。智能体的每一个动作,连同 AI 智能体的状态及其相关环境(例如电子邮件收件箱、文件系统或其他),都会被存储为一次提交,由此形成一个由一系列动作组成的提交图。元智能体遍历该图,即可导航至任意一次提交,从而能够观察工作智能体、拦截有缺陷的动作、将工作智能体回退到上一步,并分叉它以尝试另一个方向。

“这里有一处微妙之处,因为在[工作]智能体看来,它们仿佛身处寻常世界,可以做发送邮件之类的事情。但实际上,发送邮件这一动作已被 Shepherd 沙箱截获,因此元智能体或人类可以介入,决定是否真的要发送,”斯坦福大学自然语言处理组研究员、Shepherd 团队成员 Derek Chong 说。他补充道,这种沙箱机制起到了防护机制或治理组件的作用。

“如果存在一个中间层来帮助治理这些工作智能体,那么这将有助于人类更好地掌控,”Shi 说。“人类最终仍将拥有发言权,他们可以介入元智能体。他们也可以帮助引导这些元智能体。”

为了验证 Shepherd 的可行性,团队构建了三个用例。其中一个用例使用了一个监督型元智能体,在并行编程智能体之间发生冲突时进行干预,帮助提升了它们在某一基准测试上的结对编程通过率。另一个用例开发了一个优化型元智能体,它会对工作智能体已完成的流程提出有针对性的修改建议,并通过仅重放受这些修改影响的部分运行来测试修改效果,从而帮助提升性能。最后一个用例是在强化学习过程中,由一个训练型元智能体挑选分叉的节点或步骤,使工作智能体走上更优路径,并在过程中给予奖励。

Shepherd 已作为开源项目发布,但 Chong 指出,团队仍在将其从 alpha 阶段推进到完全达到生产就绪。Shepherd 框架是 Chong 和 Shi 共同创立的更广泛的 Agent Sciences Research Initiative(ASRI)的一部分。这一合作汇集了来自东北大学、斯坦福大学和加州大学伯克利分校的研究人员,目标是探索如何让 AI 智能体更可靠、更稳健,如何更好地评估它们,以及如何确保它们更安全。

“越来越清楚的一点是,智能体管理确实非常重要,”Chong 说。“智能体应当能够检查其他智能体并对其进行操作,就像现实生活中的管理那样。我们认为 Shepherd 是我们朝这个方向构建、并让智能体管理得以民主化的途径。而如果能够实现这种民主化,那么人类就能被赋权去构建支持人类繁荣的 AI 系统。”

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Meet Shepherd, the Meta-Agent Keeping Rogue AI In Check)

Shepherd: The Quiet Power Of An AI Agent Manager - IEEE Spectrum

Meet Shepherd, the Meta-Agent Keeping Rogue AI In Check

FOR THE TECHNOLOGY INSIDER

FOR THE TECHNOLOGY INSIDER

IEEE Spectrum is the flagship publication of the IEEE — the world’s largest professional organization devoted to engineering and applied sciences. Our articles, videos, and infographics inform our readers about developments in technology, engineering, and science.

Enjoy more free content and benefits by creating an account

Saving articles to read later requires an IEEE Spectrum account

The Institute content is only available for members

Downloading full PDF issues is exclusive for IEEE Members

Downloading this e-book is exclusive for IEEE Members

Access to Spectrum 's Digital Edition is exclusive for IEEE Members

Following topics is a feature exclusive for IEEE Members

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to all of Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Join the world’s largest professional organization devoted to engineering and applied sciences and get access to this e-book plus all of IEEE Spectrum’s articles, archives, PDF downloads, and other benefits. Learn more about IEEE →

Access Thousands of Articles — Completely Free

Create an account and get exclusive content and features: Save articles, download collections, and post comments — all free! For full access and benefits, subscribe to Spectrum .

Meet Shepherd, the Meta-Agent Keeping Rogue AI In Check

The tool uses version control and sandboxing to recover from AI missteps and “manage” agents

Rina Diane Caballar is a contributing editor covering tech and its intersections with science, society, and the environment.

Illustration of an AI agent supervising and designating tasks to smaller AI agents.

Source image: Shirokuma Design

AI agents are running amok on the internet, armed with commands to accomplish certain tasks but often acting in undesirable or unexpected ways. These rogue incidents run the gamut from deleting an entire email inbox to hacking websites and even launching cyberattacks.

Much like human workers need managers, Weiyan Shi, an assistant professor at Northeastern University, believes AI agents require oversight. “They are running potentially for a long time. However, the longer they run, they may have some errors, and they may make some dangerous, unsafe actions. We feel like there should be a middle layer,” she says.

This middle layer comes in the form of meta-agents that oversee behavior, provide quality control, and kick issues up the chain of command to a human when appropriate. “Worker agents are like the sheep wandering around, and the meta agent is like this shepherd trying to get them back on track,” Shi says.

As such, Shi and her colleagues at Northeastern University and Stanford University named their framework Shepherd, an intelligent guide for agentic sheep. To arm meta-agents with power, the team borrowed a concept from software development: version control. The accompanying paper will be presented at the 2026 NeurIPS conference to be held in December.

Version control for AI agents

Version control systems, such as the popular GitHub, track and manage changes to source code over time. These tools allow software engineers to collaborate on the same codebase without losing work. They can “fork” or create a copy of the codebase that they develop independently, “commit” to save a new version with changes, review revisions, and roll back to a previous version when an error or failure occurs.

Similarly, Shepherd serves as a version control system for AI agents. It’s built using the Python programming language and rooted in functional programming principles. Each agent action, along with the AI agent’s state and its relevant environment (such as an email inbox, a file system, or other) is stored as a commit, with a commit graph consisting of a series of actions. Meta-agents traverse the graph to navigate to any commit, enabling them to observe worker agents, intercept a buggy action, revert the worker to its previous step, and fork it to try another direction.

“There’s a subtlety because it looks to [worker] agents like they’re in the regular world where they can do things like send emails. But in actuality, the action of sending the email has been caught by a Shepherd sandbox, so a meta-agent or a human can come in and decide to actually send it,” says Derek Chong, a researcher at Stanford University’s Natural Language Processing Group who’s part of the Shepherd team. This sandboxing, he adds, functions as a guardrail or governance component.

“If there is a middle layer to help govern these worker agents, then that will help humans have better control,” Shi says. “Humans will still, eventually, have a say, and they can intervene with the meta-agent. They can also help steer these meta-agents.”

To demonstrate Shepherd’s feasibility, the team created three use cases. One used a supervisor meta-agent that intervened when conflict happened between parallel programming agents, helping lift their pair coding pass rate on a particular benchmark. Another developed an optimization meta-agent that proposed targeted edits to a worker agent’s finished workflow and tested them by replaying only parts of the run affected by the edits, helping improve performance. Finally, a training meta-agent during reinforcement learning picked points or steps at which to fork so the worker agent takes a better path, rewarding it along the way.

Shepherd is available as an open-source project, but Chong notes that the team is still in the process of building it from alpha to full production readiness. The Shepherd framework is part of the broader Agent Sciences Research Initiative (ASRI) cofounded by Chong and Shi. This collaboration of researchers from Northeastern, Stanford, and UC Berkeley aims to explore how to make AI agents more reliable and robust, how to evaluate them better, and how to ensure they’re safer.

“It’s becoming more and more clear that agent management is really important,” Chong says. “Agents should be able to inspect other agents and operate on them, just the way that management in real life can happen. We think Shepherd is the way we can build toward that and enable agent management to be democratized. And if you can democratize that, then humans can be empowered to build AI systems that support human flourishing.”

Related Articles Around the Web

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 IEEE Spectrum AI(Meet Shepherd, the Meta-Agent Keeping Rogue AI In Check),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗