我们如何打造构建 AWS Emulator 的智能体

AWS Emulator 是一个 AWS 服务的本地模拟器,100% 由自主 AI 智能体编写。
数千个自主智能体编写测试、发现问题并修复代码,全程无需人工审查。
源代码以 CC0 许可证在 GitHub 上提供,并且持续更新。
我是个糟糕的画家。一直都是。我父母让我提前一年上了幼儿园。有一天,老师让全班同学坐到一张长桌旁。她把一大卷纸铺在桌上,让我们画一幅画,描绘有恐龙时地球的样子。我开始认真地画属于我的那部分画面。但是,作为班里年龄最小的孩子之一,我的运动能力还不足以正确握住蜡笔。几分钟后,我的同学冲我大喊大叫。我全画错了。
今年,我经常想起那些同学。就像他们对我发脾气一样,我发现自己对我那些 AI 智能体爆粗口的次数,多到我不好意思承认。但这两件事的问题都不在于恶意。智能体并不是想违抗我,就像我也不是想毁掉班里的恐龙画一样。这是能力不足。今年早些时候,我的伴侣买了一本涂色书,我们一起花了几个小时涂完了一页。被给予一个界定清晰的区域去填充,并且知道更大的画作终会拼合而成,这让人感到某种安心。我觉得智能体也需要涂色书。
今天,我们发布了 AWS Emulator,这是一个 AWS 服务的本地模拟器,你可以在自己的机器上运行它来构建和测试 AWS 应用。该模拟器完全由自主智能体编写。我们团队没有编写或审查任何代码。我们能这样做,是因为我们设置的防护机制给了智能体独立工作的空间,也让我们相信它们不会越界。
AWS Emulator 始于一次周末实验
今年早些时候,Anthropic 的研究人员证明,AI 智能体可以自主构建一个 C 编译器。这让我灵光一现:如果给定定义明确的规范和参考实现,智能体就能自主构建出忠实实现该规范的解决方案。AWS 两者都具备;AWS API Models 是我们的规范,而实际的 AWS 云是我们的参考实现。为了验证我的假设,我给 Kiro 输入了我能想到的最简单的提示词,完全没有指望这能成功:
基于 AWS Command Line Interface (CLI) 编写测试
不断迭代这些测试,直到它们在真实 AWS 上通过
构建一个内存中实现,使同样的测试能在其上通过
15 分钟后,Kiro 带着一个还算过得去的 Amazon Simple Queue Service (Amazon SQS) 初版回来了。我优化了提示词,让 Kiro 在周末为我构建 AWS 的其余部分。它做到了。它显然还有缺口,但已经足够有说服力,以至于我的团队那个周一就开始用它来测试我们的软件。
要把它变成一个稳健的产品,花的可不只是一个周末的功夫。如今人们很容易看到这样的成果,就相信 AI 什么都能造。也许最终确实如此。但要做出让你有信心交到客户手里的东西,仍然需要实打实的工程投入。就 AWS Emulator 而言,这需要构建防护机制,确保智能体在我们划定的边界内行事。
三支 AI 智能体团队如何构建 AWS Emulator
AWS Emulator 从根本上是靠三支智能体团队构建的。一支团队管理测试套件;一支团队发现模拟器中的问题;一支团队修复这些问题。我们把智能体的职责划分成这几支团队,是因为我们发现它们在边界明确的任务上成功率更高。
一张图示,展示多支 AI 智能体团队如何协同工作,运行测试并解决问题。
第一支团队的测试套件衡量的是与真实 AWS 云的保真度。每个测试都是一个真实世界的 AWS 场景。例如,针对 Amazon SQS 的测试可能包括创建队列、发送消息、接收消息、确认消息并删除队列。智能体根据公开发布的 AWS API Models 设计这些测试,并在提交前逐一对照真实 AWS 进行验证。我们在整个 AWS 范围内拥有数万个保真度测试,而且随着智能体发现它们能用 AWS 做的新事情,这套测试套件还在不断扩展。这些测试是我们系统中最根本的防护机制之一,因为它们为智能体提供了一个必须达成的、明确无歧义的目标,并且可以反复重跑,而无需再次调用 AWS。
第二组智能体重放保真度测试,以找出仿真器中的差异。当系统最初创建该代码仓库时,它生成了一个空的 Web 服务器,能够处理 438 个 AWS API 模型的请求,但什么也不做。智能体们涌入代码仓库,找出哪些地方不工作。为了启动这一过程,最初的问题描述很简单,只写着“构建 Amazon SQS 的第一个版本”。一旦该服务完成引导,智能体就会重放保真度测试,并针对失败的测试提交问题。它们被要求对失败原因进行推理,并将可能具有相同根因的失败归为一组。
每个问题交给第三组中的智能体,其职责是修复这一个问题。它们检出代码,定位问题的根因,并编辑代码直至问题得到解决。之后,它们必须让代码改动通过一道关卡,以确保其达到我们的标准。这道关卡显然会拦截无法编译的改动。但它还包括一些智能体,负责对照编码标准进行审查、验证代码是否安全,以及确保该改动没有导致保真度回退。如果其中任何一项检查失败,改动就会被退回给该智能体返工;如果全部通过,就会被合并到主干。我们每天看到成千上万个问题被修复,数百个智能体同时在代码库上工作。如果没有这道合并关卡,我们就不会信任智能体所构建的东西。我们投入大量精力对其进行优化,因为它对智能体构建的速度和可靠性至关重要。
AWS Emulator 如何发布到 GitHub
一旦合并到主线,这些改动就会流经一条完全构建在 AWS Developer Tools 之上的发布流水线,其中 AWS CodeCommit 是这些智能体的主要代码仓库。提交通常会堆叠在一起,我们会先将它们一起构建并重新测试,然后再合并到发布分支。随后,我们把模拟器打包成一个可下载的二进制文件。只有到发布流程的最后,我们才会把代码也发布到 GitHub 上。
三个 AI 智能体团队构建了 AWS Emulator。
我们把代码发布到 GitHub,这样你也可以看到这些智能体在数千次提交中构建这个模拟器都做了些什么。提交的每一个 issue 都在仓库里。它们已经修复的放在“issues/things-that-now-work”目录下;积压未处理的放在“issues/things-that-do-not-work”目录下。如果你深入挖掘,会发现一些很有意思的东西。在不断完善构建机器的过程中,我们的智能体已经构建了这个模拟器的许多版本。我个人最喜欢的是较早版本中的一个:当时它们决定自己构建一个 Python 解释器来支持 AWS Lambda 函数。此后我们已经给它们提供了复用现有库的指导,这对不需要涂色书的人来说似乎是显而易见的事。我很期待在这个版本中智能体还会提出哪些新颖的解决方案。但因为有防护机制,我对结果有信心。我们从这段历史中获得的洞见,是我们决定以 Creative Commons CC0 许可协议将此软件作为公开奉献发布的主要动机。
AWS Emulator 的下一步
这是一系列文章中的第一篇,我将深入探讨诸如我们如何设计多智能体工作流、如何衡量保真度,以及如何确保我们的质量和安全标准等话题。
源代码和保真度测试都可以在 GitHub 上获取。
热门新闻与故事
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(How we built the agents that built AWS Emulator)
AWS Emulator is a local emulator of AWS services, 100% written by autonomous AI agents.
Thousands of autonomous agents write tests, find issues, and fix code without human review.
The source code is available on GitHub under a CC0 license and is constantly being updated.
I’m a terrible artist. I always have been. My parents sent me to kindergarten a year early. One day, the teacher sat the class down at one of those long tables. She stretched a giant roll of paper across it and asked us to draw a picture of what the Earth must have been like with dinosaurs. I began diligently working on my section of the scene. But, as one of the youngest in the class, my motor skills were not developed enough to hold a crayon correctly. A few minutes later, my classmates were screaming at me. I had done it all wrong.
I thought about those classmates often this year. Just as they lost their temper with me, I’ve found myself swearing at my AI agents more times than I care to admit. But in neither case is the issue malintent. The agents aren’t trying to disobey me, just as I wasn’t trying to ruin my class’s dinosaur painting. It’s a lack of competence. Earlier this year, my partner bought a coloring book, and we spent hours doing one together. There was something comforting about being given a well-defined area to fill in, knowing that the larger piece of art would come together. I think agents also need coloring books.
Today, we released AWS Emulator, a local emulator of AWS services that you can run on your machine to build and test your AWS applications. The emulator was written entirely by autonomous agents. Our team hasn’t written or reviewed any of the code. We could do this because the guardrails we’ve put in place gave the agents space to work independently and us the trust that they don’t color outside these lines.
AWS Emulator started as a weekend experiment
Earlier this year, researchers at Anthropic demonstrated that AI agents could autonomously build a C compiler. This was a lightbulb moment: If given a well-defined spec and a reference implementation, agents can autonomously build a solution that faithfully implements the specification. AWS has both of those; the AWS API Models are our spec and the actual AWS cloud is our reference implementation. To test my hypothesis, I gave Kiro the simplest prompt that I could think of without any expectation that this would work:
Write tests based on the AWS Command Line Interface (CLI)
Iterate on those tests until they pass against real AWS
Build an in-memory implementation that those same tests pass against
Kiro came back after 15 minutes with a passable first version of an Amazon Simple Queue Service (Amazon SQS). I refined my prompt and had Kiro build me the rest of AWS over the weekend. And it did. It obviously had gaps, but it was compelling enough that my team started using it to test our software that Monday.
Turning that into a robust product took more than a weekend. There’s a natural temptation these days to see results like those and believe that AI can build anything. Maybe it will eventually. But building something that you feel confident putting in your customers’ hands still requires serious engineering effort. In the case of AWS Emulator, it required building guardrails that ensure the agents color within our lines.
How three teams of AI agents build AWS Emulator
AWS Emulator is fundamentally built by three teams of agents. One team manages a test suite; one team finds issues in the emulator; and one team fixes those issues. We divide agent responsibility into these teams because we’ve found they have more success on bounded tasks.
A graphic showing how teams of AI agents work together to run tests and solve issues.
The first team’s test suite measures fidelity against the real AWS cloud. Each test is a real-world AWS scenario. For example, a test for Amazon SQS might include creating a queue, sending a message, receiving it, acknowledging it, and deleting the queue. Agents devise these tests from publicly released AWS API Models and verify each one against real AWS before submitting. We have tens of thousands of fidelity tests across all of AWS, and that test suite constantly grows as our agents discover new things that they can do with AWS. These tests are one of our most fundamental guardrails in our system because they provide an unambiguous goal that the agents need to meet and can be rerun repeatedly without calling AWS again.
The second team of agents replays the fidelity tests to find divergence in the emulator. When the system initially created the repository, it generated an empty web server that could handle requests for the 438 AWS API Models but did nothing. The agents swarm the repository to figure out what doesn’t work. To kickstart the process, the first issues simply say “build a first version of Amazon SQS.” Once the service is bootstrapped, the agents replay the fidelity tests and file issues for the failing tests. They’re told to reason about the failures and group failures that may share a root-cause.
Each issue is given to an agent in the third team, whose job is to fix that single issue. They check out the code, root-cause the issue, and edit the code until it’s addressed. They must then pass their code change through a gate that ensures it meets our bar. This gate obviously blocks changes that don’t compile. But it also includes agents that review against coding standards, verify the code is secure, and ensure that the change did not regress fidelity. If any of those checks fail, the change is sent back to the agent for rework; if they all pass, it is merged to mainline. We’ve seen thousands of issues fixed every day as hundreds of agents concurrently work on the codebase. Without this merge gate, we wouldn’t trust what our agents are building. We focus a lot on optimizing it because it’s so critical to how quickly and reliably the agents build.
How AWS Emulator ships to GitHub
Once merged to mainline, the changes flow through a release pipeline built entirely on AWS Developer Tools, with AWS CodeCommit as the agents’ primary code repository. Commits typically stack up together and are built and retested before we merge them to a release branch. We then package up the emulator as a binary that you can download. Only at the end of the release process do we also publish the code to GitHub.
Three teams of AI agents build AWS Emulator.
We release the code to GitHub so that you can also see what the agents have done building this emulator over thousands of commits. Every issue filed is in the repository. The ones they’ve fixed are in the directory “issues/things-that-now-work”; the ones in their backlog are in “issues/things-that-do-not-work.” You’ll find fascinating stuff if you dig. We’ve had our agents build many versions of this emulator as we’ve refined the building machine. My personal favorite was from an earlier version, when they decided to build their own Python interpreter to support AWS Lambda functions. We’ve since given them guidance to reuse existing libraries, something that may seem obvious to someone who doesn’t need a coloring book. I’m looking forward to what other novel solutions the agents come up with in this version. But I trust the outcome because of our guardrails. The insights we’ve gained from this history were the main motivation in our decision to release this software as a public dedication under the Creative Commons CC0 license.
What's next for AWS Emulator
This is the first in a series of posts where I’ll dive deeper into topics such as how we architected our multi-agent workflow, how we measure fidelity, and how we ensure our quality and security bar.
The source code and fidelity tests are both available on GitHub.
Trending news and stories
原文为英文,正文由自动抓取并保留原文。
原文发布于 About Amazon(How we built the agents that built AWS Emulator),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。