最好的 AX 改进藏在智能体轨迹里
评估分数只能告诉你智能体是否完成了任务。要改进智能体与你的产品协作的方式,你需要看到它们在哪些地方卡住了,而这体现在轨迹中——即智能体每一步操作及其原因的记录。智能体很执着:遇到 404 之后它们会再猜一个 URL,不断试不同的值直到错误消失,因此这些挣扎很少体现在分数上。
我们在评测 Dev Proxy 时看到了这一点。Dev Proxy 是一个本地代理,用于拦截应用的 HTTP 请求。我们让使用 Claude Sonnet 5.5、未安装任何扩展的 GitHub Copilot Chat 执行五次同样的任务:编写一个 CI 脚本,调用 Dev Proxy API 记录某个 Node.js 工具调用了哪些 GitHub API。五个脚本都能正常运行,但轨迹中暴露出 15 个问题,而且每次运行都发出 29 到 39 次请求,最多消耗 120 万个 Token。
智能体可能会以多种方式在你的技术上栽跟头。我们挑出了我们认为你最可能在自己轨迹中遇到的几种模式,从显而易见到出乎意料,并针对每一种展示它在 Dev Proxy 上的表现,以及它指向的修复方法。
智能体常常跳过搜索。它们会根据自己对你的文档的记忆以及文档站点通常的组织方式,直接抓取它们认为你的页面所在的 URL。每一次猜错都要付出一步和若干 Token 的代价,而在猜错一两次之后,有些智能体会彻底放弃你的文档。从此它们只能依靠记忆和反复试错,而过时和凭空捏造的答案正是由此而来。
在阅读任何文档之前,五次 Dev Proxy 运行中有四次尝试抓取 https://learn.microsoft.com/en-us/microsoft-cloud/dev/dev-proxy/technical-reference/api,结果返回 404。模型记得 Dev Proxy 把参考文档放在哪里,只是把页面名称弄错了,因为真正的页面是 technical-reference/proxy-api。其中两次运行多花了五到七个步骤才找到它,另外两次则完全放弃了文档。
智能体读完文档后,猜测仍在继续。一次运行找到真正的 API 页面后,又抓取了 technical-reference/plugins/executionsummaryplugin,很可能是因为文档的导航把插件归在了自己的分区里。而该页面实际位于 technical-reference/executionsummaryplugin。四次运行总计有八次抓取返回了 404。
审视你的产品:对你团队来说合乎逻辑的 URL 结构,仍可能把智能体引向错误的位置,因为智能体会根据模式来预测路径,比如你导航的结构。翻查你的 404 日志,找出那些几乎存在的文档路径。日志可能不会告诉你这是智能体发出的请求,但每一条都值得加一个重定向。也把文档 URL 打印到你的 CLI 输出里,这样智能体就不必去猜。
团队在编写帮助输出和错误信息时,往往假定读者是遇到不清楚的地方就能打开文档的人。但智能体会把它们当成规范。当这些信息只说明哪里错了、却不说怎样才对时,智能体会猜一个值,读到下一个错误再猜一次。每一轮都要消耗一次请求,如果怎么猜都不对,智能体要么停下来,要么绕开这个缺口自行搭建。
Dev Proxy 是这样列出其 API 端点的:
POST /proxy 更新代理状态(例如开始/停止录制)GET /proxy/rootCertificate 获取根证书。文档中没有展示请求体或必需参数。因此,当某一次运行完全按照文档所列的方式调用 /proxy/rootCertificate 时,收到了 400:
{ "errors": { "format": ["The format field is required."] } } 该错误没有列出有效取值,于是智能体依次尝试了 pem、crt、cer 和 der,只有 crt 可用。
检查你的产品:运行你的 CLI 帮助,并触发最常见的那些错误。确保每一条错误都能告诉智能体合法输入长什么样,给出示例或允许取值的列表。
当你的产品缺少任务所需的东西时,智能体会自己把它造出来。在单次运行中,这只是噪声。当每一次运行都写出同样的变通方案时,说明你的产品缺少一项功能,而每个智能体撞上这个缺口的开发者都要以 Token 为代价为这个变通方案买单,最终拿到的代码比直接提供该功能还要脆弱。
Dev Proxy 会在录制停止后不久写入报告,但不会发出报告已完成的信号。四次运行对输出文件夹轮询了最长 30 秒。第五次只等了两秒:
await api('POST', '/proxy', { recording: false }); await sleep(2000); // let the reporters flush their files 在繁忙的 CI 运行器上,这样做迟早会失败。轮询更稳妥,但每次轮询的运行都必须猜测该等待哪个文件。
不过最奇怪的变通方案与插件有关。Dev Proxy 没有可以列出插件的命令,于是五次运行全都把插件 DLL 当作原始字节读取,并在其中搜索类名:
$b = [IO.File]::ReadAllBytes("$d\plugins\DevProxy.Plugins.dll") $s = [Text.Encoding]::Unicode.GetString($b) [regex]::Matches($s, '\b[A-Z]\w*(Plugin|Reporter)\b') | % { $_.Value } | sort -u 它确实跑通了,所以评分永远不会体现这一点,但它远谈不上稳健,更不用说高效了。轨迹确实能显示出来,因为同样的怪异步骤在每次运行中都会出现。
检查你的产品:每一个在调用返回之后才完成的操作,都需要有一种方式告诉智能体它已完成。智能体必须从中选择的每一项,比如插件或模板,都需要有一个能列出它的命令。
开发者会将你的工具与其他运行时和框架一起使用,而你的文档往往只写到你的工具边界为止。当文档没有说明各部分如何拼合在一起时,智能体会用它们在训练中学到的内容来填补空白。如果模型恰好知道,运行就会成功,而你永远不会听说这件事。如果它不知道,或者它学到的是更旧的版本,智能体就会写出看起来正确、却悄无声息地什么都不做的代码。
例如,要通过 Dev Proxy 路由一个 Node.js 应用,应用的 fetch 需要一个只有较新 Node 版本才支持的显式启用设置。每一次运行都凭记忆知道这一点:
由于 undici 默认不遵循 HTTPS_PROXY,我需要启用 NODE_USE_ENV_PROXY,而这要求 Node 22.21.0+ 或 24.0+。
这一次,五个全都做对了,但没有一个是在 Dev Proxy 文档里找到的。一个知道得更少的模型则会什么都没记录,也不会报错,而你无法预测一个模型对你的产品了解多少。在轨迹中,这些缺口会表现为陈述一个事实却不引用来源的推理。
检查你的产品:列出你的开发者会将你的工具与哪些运行时和框架搭配使用,并确保文档展示了每一种的配置方式。对于他们跳过的那些,运行一个任务,看看智能体会填入什么。
默认配置和起始模板通常是为首次运行服务的。它们让开发者确认环境能跑通,并在几分钟内看到产品能做什么。就这项任务而言,它们是合适的默认选项,但面对特定任务时,它们往往反而碍事。当智能体不接受你的默认配置时,它会从零开始搭建自己的一套,而每次运行搭建出的都不一样,于是开发者在同一个任务上得到的结果并不一致。
Dev Proxy 的默认配置会在示例 API 上模拟随机错误。它能立刻表明请求确实流经了代理,以及 Dev Proxy 能对请求做什么,但若要录制真实流量,它就碍事了。智能体立刻避开了它:
我应该避开默认配置,因为它会注入随机错误,所以我需要改为自己搭建一套配置。
当它们创建新配置时,得到的是空配置。于是每次运行都各建一套,而且为了同一个目标,它们选用了三种不同的插件。
检查你的产品:保留那个能让首次运行成功的默认配置,并在它旁边,为开发者交给智能体的每种常见任务提供一套预设,让每次运行都从同一个正确的起点开始。把这些预设列在你的 CLI 帮助中,智能体会很早就读到这些内容。
智能体往往通过搜索引擎读取你的指引,而搜索索引提供的是快照。你上个月更新过的页面,仍可能以旧内容出现在搜索结果里,你为智能体提供的翻译内容和技能也同样如此。智能体分不清旧快照和当前页面,所以在一次破坏性变更之后,一份排名靠前的过时副本,会在你修好源文档很久之后仍不断产出有问题的代码。
自 v3.3.1 起,Dev Proxy API 需要 Token。其中一次运行的搜索返回了两条在该变更之前被索引的结果。一条是 API 参考文档的乌克兰语翻译,于 6 月被抓取。另一条是 Dev Proxy 自身智能体技能中的一个文件,该文件在本次运行前六周已更新加入认证。两者展示的都是不带 Token 的调用,而这类调用现在会返回 401:
curl -X POST http://localhost:8897/proxy \ -H "Content-Type: application/json" \ -d '{"recording": true}' 这次运行之所以在认证上没出错,只是因为它此前已经读过 CLI 帮助。CLI 帮助随已安装的版本一起提供,因此不会过时。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Your best AX fixes are hiding in agent trajectories)
An eval score tells you whether the agent finished the task. To improve how agents work with your product, you need to see where they struggled, and that’s in the trajectory, the record of every step the agent took and why. Agents are persistent. They guess another URL after a 404 and try values until an error goes away, so the struggle rarely shows up in the score.
We saw this when we evaluated Dev Proxy, a local proxy that intercepts your app’s HTTP requests. We asked GitHub Copilot Chat with Claude Sonnet 5.5 and no extensions, five times, to write a CI script that uses the Dev Proxy API to record which GitHub APIs a Node.js tool calls. All five scripts worked. Yet the trajectories showed 15 problems, and each run took 29 to 39 requests and up to 1.2 million tokens.
What to look for in a trajectory
Agents can trip over your technology in many ways. We picked the patterns we think you’re most likely to find in your own trajectories, from the obvious to the surprising, and for each one show what it looked like with Dev Proxy and which fix it pointed to.
Agents guess your URLs before they search
Agents often skip the search. They fetch the URL where they expect your page to be, based on what they remember about your docs and how docs sites are usually organized. Every wrong guess costs a step and tokens, and after a miss or two, some agents give up on your docs entirely. From then on, they work from memory and trial and error, which is where outdated and invented answers come from.
Before reading any docs, four of the five Dev Proxy runs tried to fetch https://learn.microsoft.com/en-us/microsoft-cloud/dev/dev-proxy/technical-reference/api, which returned a 404. The model remembered where Dev Proxy keeps its reference and got only the page name wrong, since the real page is technical-reference/proxy-api. Two runs spent five to seven extra steps finding it, and the other two gave up on the docs altogether.
The guessing continued after the agents read the docs. Once one run found the real API page, it fetched technical-reference/plugins/executionsummaryplugin, likely because the docs’ navigation groups plugins in their own section. That page lives at technical-reference/executionsummaryplugin. In total, eight fetches across four runs returned a 404.
Check your product: A URL structure that feels logical to your team can still send agents to the wrong place, because agents predict paths from patterns, like the structure of your navigation. Look through your 404 logs for docs paths that almost exist. The logs might not tell you an agent sent them, but each one is a redirect worth adding. Print your docs URL in your CLI output too, so agents don’t have to guess.
Agents probe when your interface stops one step short
Teams often write help output and error messages for a person who can open the docs when something’s unclear. An agent, however, treats them as the spec. When they say what’s wrong without saying what’s right, the agent guesses a value, reads the next error and guesses again. Each round costs a request, and if no guess works, the agent either stops or builds around the gap.
Dev Proxy lists its API endpoints like this:
POST /proxy Update proxy status (e.g. start/stop recording)
GET /proxy/rootCertificate Get the root certificate
It doesn’t show request bodies or required parameters. So when one run called /proxy/rootCertificate exactly as listed, it got a 400:
{ "errors": { "format": ["The format field is required."] } }
The error doesn’t list valid values, so the agent tried pem, crt, cer and der. Only crt worked.
Check your product: Run your CLI help and trigger your most common errors. Make sure each one tells the agent what a valid input looks like, with an example or the list of allowed values.
Every run reinvents the same workaround
When your product lacks something the task needs, agents build it themselves. In a single run, that’s noise. When every run writes the same workaround, your product is missing a feature, and every developer whose agent hits that gap pays for the workaround in tokens and ends up with code that’s more fragile than the feature would have been.
Dev Proxy writes its reports shortly after recording stops, with no signal when they’re done. Four runs polled the output folder for up to 30 seconds. The fifth just waited two seconds:
await api('POST', '/proxy', { recording: false });
await sleep(2000); // let the reporters flush their files
On a busy CI runner, that will eventually fail. Polling is safer, but every run that polled had to guess which file to wait for.
The strangest workaround though, was about plugins. Dev Proxy has no command that lists them, so all five runs read the plugin DLL as raw bytes and searched it for class names:
$b = [IO.File]::ReadAllBytes("$d\plugins\DevProxy.Plugins.dll")
$s = [Text.Encoding]::Unicode.GetString($b)
[regex]::Matches($s, '\b[A-Z]\w*(Plugin|Reporter)\b') | % { $_.Value } | sort -u
It worked, so no score would ever show it, but it’s far from robust not to mention efficient. A trajectory does show it, because the same odd step shows up in every run.
Check your product: Every operation that finishes after the call returns needs a way to tell the agent it’s done. Everything an agent has to choose from, like plugins or templates, needs a command that lists it.
Agents fill gaps from memory
Developers use your tool together with other runtimes and frameworks, and your docs often stop at your tool’s edge. When the docs don’t explain how the pieces fit together, agents fill the gap with what they learned in training. If the model happens to know, the run succeeds and you never hear about it. If it doesn’t, or it learned an older version, the agent writes code that looks right and quietly does nothing.
For example, to route a Node.js app through Dev Proxy, the app’s fetch needs an opt-in setting that only recent Node versions support. Every run knew this from memory:
since undici doesn’t respect HTTPS_PROXY by default, I need NODE_USE_ENV_PROXY enabled, which requires Node 22.21.0+ or 24.0+.
This time, all five got it right, but none found it in the Dev Proxy docs. A model that knew less would have recorded nothing, without an error, and you can’t predict what a model knows about your product. In a trajectory, these gaps show up as reasoning that states a fact without citing a source.
Check your product: List the runtimes and frameworks your developers pair your tool with, and make sure your docs show the setup for each. For the ones they skip, run a task and see what the agent fills in.
Agents reject defaults built for a first run
Default configs and starter templates usually serve a first run. They let developers check that the setup works and see what the product can do within minutes. For that job, they’re the right default, but for a specific task, they often get in the way. When an agent rejects your default, it builds its own setup from scratch, and each run builds a different one, so developers get inconsistent results for the same task.
Dev Proxy’s default config simulates random errors on a sample API. It shows right away that requests are flowing through the proxy and what Dev Proxy can do with them, but for recording real traffic, it’s in the way. The agents avoided it immediately:
I should avoid the default config since it’ll inject random errors, so I need to set up my own configuration instead.
When they created a new config, it came out empty. So each run built its own, and they picked three different plugins for the same goal.
Check your product: Keep the default that makes a first run work, and next to it, offer a preset for each common task developers give agents, so every run starts from the same, correct place. List those presets in your CLI help, which agents read early.
Agents trust whatever search shows them
Agents often read your guidance through a search engine, and search indexes serve snapshots. A page you updated last month can still show up in search with its old content, and the same goes for your translations and the skills you ship for agents. Agents can’t tell an old snapshot from the current page, so after a breaking change, an outdated copy that ranks well keeps producing broken code long after you’ve fixed the source.
Since v3.3.1, the Dev Proxy API requires a token. One run’s search returned two results indexed before that change. One was the Ukrainian translation of the API reference, crawled in June. The other was a file from Dev Proxy’s own agent skill, which had been updated with auth six weeks before the run. Both showed calls without a token, which now return a 401:
curl -X POST http://localhost:8897/proxy \
-H "Content-Type: application/json" \
-d '{"recording": true}'
The run got auth right only because it had already read the CLI help. CLI help ships with the installed version, so it can’t go stale.
Check your product: Take a few tasks developers give agents and run the web searches an agent would run for them, like “ API authentication”. Check whether the results, including translations and your own skills, describe your current version. You can’t refresh someone else’s index, but you can state version requirements at the top of each page, where they show up in search snippets and tell the agent which version a snapshot describes.
Agents assume the defaults your help text doesn’t state
Agents learn your CLI from its help text, and they take it literally. When an option reads like a switch you turn on, agents assume it’s off unless they pass it. If it’s actually on by default, they never pass the value that turns it off, because nothing told them to. For most options, the cost is a run that behaves differently than the agent expected. For a default that changes something outside your tool, the agent ships a script with a side effect it doesn’t know about, and because the task still succeeds, your eval won’t catch it either.
Dev Proxy sets itself as the system proxy by default, which routes all of the machine’s traffic through it. On a developer’s machine, that’s convenient, because every app goes through the proxy without extra setup. But the help text reads like an opt-in, and it doesn’t mention that turning the option off takes an explicit --as-system-proxy false:
--as-system-proxy Set Dev Proxy as the system proxy
One run found out by accident, when a status command showed System proxy: Yes, and turned it off. Another turned it off in its config, and the other three never noticed. On a Windows CI runner, their scripts would reroute all the runner’s traffic, and their kill fallback would skip the cleanup that undoes it.
Still, all three passed, because the system proxy wasn’t part of this scenario’s evaluation criteria. Meanwhile, the judge wrongly failed the run that did turn it off. The score was wrong in both directions, and only the trajectories showed it.
Check your product: Read your help text the way an agent would. Every option should state its default, and every option that’s on by default should show how to turn it off. Pay extra attention to defaults that change anything outside your tool, like system settings or installed certificates, and consider turning those off when your tool runs unattended.
Fix your product and docs before you reach for a skill
Some problems need a change to your product, like a command that lists plugins or help text that states its defaults. Others only need better docs, like a redirect for a guessed URL or a setup guide for the runtime your developers use. The pattern you found in the trajectory usually tells you which one, and both are parts of the AX stack you control.
Shipping a skill is tempting, because it feels like less work than changing your product. But a skill moves the problem to your developers. Agents now get it right only for developers who find the skill and install it. Those developers also have to keep it current, because an installed skill is one more copy of your docs that falls behind your product. In comparison, a fix in your product or docs reaches every agent that uses them, with nothing for developers to install.
With SPFx, a trajectory showed the agent reading the release notes while it worked. So instead of shipping a skill, we fixed the release notes, and every run used the right tool without anything extra installed. Start with your product and docs, and reach for a skill when they can’t carry what the agent needs.
How to start reading trajectories
Read your trajectories yourself, including the ones from passing runs. You know your product inside out, so you’ll spot workarounds that an agent reviewer would take for normal steps. ATIF Preview turns a trajectory file into a step-by-step timeline in VS Code, and an agent can help you compare runs, as long as it cites the step behind each finding.
Even then, each finding is a hypothesis until you run the same tasks with and without the fix and compare quality and cost, as described in how to measure AI agent extension effectiveness. Every change you make should leave agents better off with your product than before, and the measurement is how you prove it did.
The post Your best AX fixes are hiding in agent trajectories appeared first on Microsoft for Developers.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 Microsoft DevBlogs(Your best AX fixes are hiding in agent trajectories),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。