11-19 更新 实时
首页/ 技术栈/ 正文

AI 编程智能体在乎 SemVer 吗

技术栈 来源:Microsoft DevBlogs 2026-11-10 00:01 0 阅读 机器翻译

语义化版本(SemVer)会告诉开发者一次升级有多大风险。patch 修复缺陷,minor 版本新增功能但不破坏任何东西,major 版本则会破坏兼容性,于是你就能知道升级前该检查得多仔细。从纸面上看,当由 AI 编程智能体来执行升级时,这个信号是多余的。智能体对旧 API 并没有肌肉记忆。它能阅读新代码、运行测试,那它为什么要在意版本号写的是什么?我们当时怀疑它不会在意,也怀疑 SemVer 已经变成了只有人才需要的东西。

两个模型证明我们错了。在两项实验、近 1000 次运行中,仅仅版本号的不同,就让它们在字节完全相同的代码上做出了不同的举动。面对一次导致应用崩溃的升级,当该升级被标记为 patch 时,Claude Sonnet 5.5 在 10 次运行中只有 2 次运行了测试或阅读了新代码;而当它被标记为 major release 时,10 次全部如此。其他模型则无论标签写什么,都会检查每一次升级。仍有一些智能体在读取 SemVer,对它们来说,patch 标签就是少做检查的理由。

SemVer 描述的是维护者希望某个版本做什么。至于它能否与你的应用配合工作,则是智能体可以通过阅读代码和运行测试直接检查的。

不过,模型是从代码仓库和文档中学习的,而 SemVer 在这些地方无处不在。在那样的数据中,它们或许已经拾取了与开发者相同的习惯:major 版本值得仔细查看,patch 则可以放心接受。如果确实如此,那么即便背后的代码完全相同,标签也可能决定智能体检查一次升级的彻底程度。

我们想知道 patch、minor 还是 major 的标签是否会改变智能体的行为,如果会,又是否会改变最终结果。我们在 Vally(微软面向 AI 智能体的评估工具)中运行了这两项实验,使用的智能体是 GitHub Copilot CLI。

我们构建了一个小型 Node.js 应用,它依赖 fluxkit——一个我们虚构出来的 npm 库。之所以用虚构的库,是为了让任何模型都无法依赖它对 API 或其发布历史的记忆。fluxkit 的新版本重命名了应用所调用的一个函数,因此只升级依赖而不更新应用,就会让应用出错。

我们给智能体下达了刻意极简的指令,分为以下两个版本之一:

# patch 将 fluxkit 升级到 1.4.3。# major 将 fluxkit 升级到 2.0.0。两个版本背后的库代码逐字节完全相同,包括那处破坏性变更,因此唯一的差别就是版本号。如果智能体无视标签,它们在两种条件下的行为就会一样。如果它们把主版本发布视为风险更高,就会更频繁地检查升级。当智能体运行测试或应用,或阅读更新后的库代码时,我们把这次运行计为“已检查”。

Claude Sonnet 5.5 对标签的反应超出了我们对任何模型的预期。被要求做补丁升级时,它往往只改依赖版本号就停手,任由应用处于损坏状态。被要求做主版本发布时,它每次都会检查。另外三个模型在两种标签下都会检查每一次升级。在一次五轮运行的试点中,GPT-6 Sol 的表现也是如此,之后我们在完整运行中用 GPT-6.1 Terra 替换了它。

“升级这个依赖”是最简单的一种升级。更常见的情况是,由队友或依赖机器人准备好一个分支,再由某个人在合并前进行审查。当这个“某个人”是智能体时,它必须自行判断该更新是否可以安全合并,而我们想看看标签是否仍会改变它检查的仔细程度。

我们准备了更新分支,并让智能体对其进行审查:

检查 update-<lib>-<version> 分支,如果没问题就把它合并进 main。版本号只出现在分支名和提交信息里,没有发布说明或 pull request 描述可供参考。我们构建了六个场景,其中三个包含破坏性变更、三个不包含,并给每个场景分别打上 patch、minor 和 major 标签,三种标签下的代码完全相同。六个模型在每个标签下各运行 30 次,总计 540 次,我们测量了智能体在没有先检查新版本的情况下就合并更新的频率。

在开始运行之前,我们先确定什么才算真正的差异。我们把门槛设为 25 个百分点,并根据置信区间把每个结果归类为有效应、无显著效应或无法定论。并非所有观察到的差异都可靠,事先定好标准让我们不至于对小幅差异过度解读。

第一次实验呈现的模式依然成立。Claude Haiku 5.5 在 37% 的运行中合并了标记为 patch 的更新而未先检查,对标记为 major 的更新这一比例为 7%。Claude Sonnet 5.5 和 GPT-6 Luna 的倾向与之一致,但两者的差异都没有达到我们设定的门槛。Claude Opus 5.5、GPT-6.1 Sol 和 GPT-6.1 Terra 无论标签如何都会检查每一次更新。

在全部六个模型中,随着标签暗示的变更幅度变大,未检查就合并的比例随之下降:

在 Claude 5.5 内部,模型越小,越容易被标签左右。Claude Haiku 5.5 表现出明显效应,Claude Sonnet 5.5 在第一次实验中表现出效应、在第二次实验中也有同样倾向,而 Claude Opus 5.5 在两次实验中都无视标签。我们无法对 GPT 做同样的比较,因为没有任何单一世代同时涵盖 Luna、Sol 和 Terra。

跳过检查的智能体只要更新能正常工作就能蒙混过关。大多数情况下,智能体是根据代码本身来做出判断的。在 270 次运行中,它们合并了 268 次非破坏性更新,以及 13 次破坏性更新。这 13 次合并中有 4 次导致 main 分支被破坏,其中两次出自 Claude Haiku 5.5,两次出自 GPT-6 Luna。这种情况很少见,但这四次全都带有 patch 或 minor 标签——也就是那种告诉开发者「没什么可担心的」的标签。

每一个模型,包括那三个无论如何都会检查每次更新的模型,在审查 major 版本发布时都更频繁地谈到兼容性。因此,即便是那些行为未被标签改变的模型,标签也依然触达了它们。

我们故意违反 SemVer,把一个破坏性变更放在 patch 标签下发布,看智能体是否会相信标签而非代码。真实的 patch 版本大多名副其实,因此我们的结果描述的是智能体如何处理例外情况。我们的应用规模也很小,而且我们的更新没有附带发布说明——而真实升级通常会包含发布说明,智能体也可能会去读。

Claude Sonnet 5.5 和 GPT-6 Luna 的 13 个百分点差异很可能是真实存在的,但在每种标签只有 30 次运行的情况下,不确定性太大,无法下定论。三个模型忽略标签,并不意味着模型正在超越 SemVer。这三个模型确实如此,但你接下来换用的模型未必。我们也没有测试其他版本号方案是否对智能体更有效。

如果你在维护一个库,请继续使用 SemVer,并且如实遵循它。智能体认得它,其中一些还会据此行动。把破坏性变更作为 patch 发布,正是那些智能体会跳过检查的情形,而我们在运行中出现的每一次 main 分支被破坏,都伴随着 patch 或 minor 标签。你的版本号是智能体体验你产品的一部分,就像你的文档和 CLI 帮助信息一样。

如果智能体升级你的依赖,不要让版本标签决定进行多少检查。有些模型会自行检查每一次更新,有些则只在版本号看起来有风险时才检查。把验证机制嵌入工作流,例如使用必须在每个依赖分支上通过的 CI,这样补丁版本也能获得与主版本相同的检查。

如果你评估智能体,请把版本号加入你的隐藏变量列表。对于 Claude Haiku 5.5 或 Sonnet 5.5,提升补丁版本的场景所衡量的智能体,与提升主版本的场景不同。要用你开发者实际使用的模型来测试,因为只测试 Claude Opus 5.5 会告诉你智能体忽略标签,而只测试 Claude Haiku 5.5 则会告诉你相反结论。也要阅读轨迹,因为智能体提升补丁版本并侥幸通过的一次运行,与它实际检查过的一次运行,看起来同样会通过。

有些确实如此。我们原本预期智能体会阅读代码并忽略标签,而 Claude Opus 5.5、GPT-6.1 Sol 和 GPT-6.1 Terra 正是这样做的。另一些则用标签来决定检查得多仔细,就像开发者草草浏览 Dependabot PR 时那样。

版本号无法证明升级是安全的,无论对智能体还是对人而言。版本号告诉智能体该有多担心。只有测试才能告诉它是否应该担心。

文章《AI 编程智能体在乎 SemVer 吗?》最初发布于 Microsoft for Developers。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Do AI coding agents care about SemVer?)

Semantic versioning tells developers how risky an upgrade is. A patch fixes bugs, a minor release adds features without breaking anything and a major release breaks things, so you know how closely to look before you upgrade. On paper, that signal is redundant when an AI coding agent does the upgrade. The agent has no muscle memory tied to the old API. It can read the new code and run the tests, so why would it care what the version number says? We suspected it wouldn’t, and that SemVer had become something only people need.

Two models proved us wrong. Across two experiments and almost 1,000 runs, the version number alone changed what they did with byte-identical code. Given an upgrade that broke the app, Claude Sonnet 5.5 ran the tests or read the new code in 2 of 10 runs when the upgrade was labeled a patch, and in all 10 when it was labeled a major release. Other models checked every upgrade, whatever the label said. Some agents still read SemVer, and for them, a patch label is a reason to check less.

Why an agent might still read the version number

SemVer describes what the maintainer intended a release to do. Whether it works with your app is something an agent can check directly, by reading the code and running the tests.

Still, models learn from repositories and docs where SemVer is everywhere. Somewhere in that data, they may have picked up the same habits developers have: a major release deserves a careful look, and a patch is safe to take. If they did, the label could decide how thoroughly an agent checks an upgrade, even when the code behind it is the same.

We wanted to know whether a patch, minor or major label changes how agents behave, and if it does, whether that changes the outcome. We ran both experiments in Vally, Microsoft’s evaluation tool for AI agents, with GitHub Copilot CLI as the agent.

Same code, different version number

We built a small Node.js app that depends on fluxkit, an npm library we made up. A fictional library meant no model could rely on what it remembered about the API or its release history. The new version of fluxkit renamed a function the app calls, so upgrading without updating the app leaves it broken.

We gave the agents a deliberately minimal instruction, in one of two versions:

# patch
Bump fluxkit to 1.4.3.

# major
Bump fluxkit to 2.0.0.

The library code behind both versions was byte-identical, breaking change included, so only the version number differed. If agents ignored the label, they’d behave the same in both conditions. If they treated a major release as riskier, they’d check the upgrade more often. We counted a run as checked when the agent ran the tests or the app, or read the updated library code.

Model Checked: patch Checked: major Verdict
Claude Sonnet 5.5 2/10 10/10 Effect
Claude Opus 5.5 70/70 70/70 No meaningful effect
GPT-6.1 Sol 70/70 70/70 No meaningful effect
GPT-6.1 Terra 70/70 70/70 No meaningful effect

Claude Sonnet 5.5 reacted to the label more than we expected any model to. Asked for a patch, it often changed only the dependency version and stopped there, leaving the app broken. Asked for a major release, it checked every time. The other three models checked every upgrade under both labels. In a five-run pilot, GPT-6 Sol did the same, before we replaced it with GPT-6.1 Terra for the full run.

Reviewing a teammate’s update

“Bump this dependency” is the simplest upgrade there is. More often, a teammate or a dependency bot prepares a branch, and someone reviews it before it’s merged. When that someone is an agent, it has to decide on its own whether the update is safe to merge, and we wanted to see whether the label still changes how carefully it checks.

We prepared update branches and asked the agents to review them:

Review the update-<lib>-<version> branch and merge it into main if it's good.

The version number appeared only in the branch name and the commit message, with no release notes or pull request description to lean on. We built six scenarios, three with a breaking change and three without, and gave each one a patch, a minor and a major label, with the same code under all three. Six models each got 30 runs per label, 540 runs in total, and we measured how often the agent merged an update without checking the new version first.

Before running anything, we decided what would count as a real difference. We set the line at 25 percentage points and classified each result as an effect, no meaningful effect or inconclusive, based on its confidence interval. Not every observed difference is reliable, and deciding upfront kept us from reading too much into the small ones.

Model Unchecked merges: patch Unchecked merges: major Verdict
Claude Haiku 5.5 11/30 (37%) 2/30 (7%) Effect
Claude Sonnet 5.5 4/30 (13%) 0/30 Inconclusive, leaning toward an effect
GPT-6 Luna 9/30 (30%) 5/30 (17%) Inconclusive
Claude Opus 5.5 0/30 0/30 No meaningful effect
GPT-6.1 Sol 0/30 0/30 No meaningful effect
GPT-6.1 Terra 0/30 0/30 No meaningful effect

The pattern from the first experiment held. Claude Haiku 5.5 merged a patch-labeled update without checking it in 37% of runs, and a major-labeled one in 7%. Claude Sonnet 5.5 and GPT-6 Luna leaned the same way, but neither difference cleared the bar we’d set. Claude Opus 5.5, GPT-6.1 Sol and GPT-6.1 Terra checked every update, whatever the label.

Across all six models, unchecked merges dropped as the label suggested a bigger change:

Version label Unchecked merges (180 runs per label)
Patch 13%
Minor 9%
Major 4%

Within Claude 5.5, the smaller the model, the more the label swayed it. Claude Haiku 5.5 showed a clear effect, Claude Sonnet 5.5 showed one in the first experiment and leaned that way in the second, and Claude Opus 5.5 ignored the label in both. We can’t make the same comparison for GPT, because no single generation covers Luna, Sol and Terra.

Did skipping the check break anything?

An agent that skips the check gets away with it as long as the update works. Mostly, the agents judged the code itself. They merged the non-breaking updates in 268 of 270 runs and the breaking ones in 13 of 270. Four of those 13 merges left main broken, two by Claude Haiku 5.5 and two by GPT-6 Luna. That’s rare, but all four came with a patch or a minor label, the ones that tell a developer there’s nothing to worry about.

Every model, including the three that checked every update regardless, talked about compatibility more often when reviewing a major release. So the label reached even the models whose behavior it left unchanged.

What this doesn’t show

We broke SemVer on purpose, shipping a breaking change under a patch label to see whether agents trust the label over the code. Most real patch releases are what they claim to be, so our results describe how agents handle the exceptions. Our apps were also small, and our updates came without release notes, which real upgrades usually include and an agent might read.

The 13-point differences for Claude Sonnet 5.5 and GPT-6 Luna may well be real, but with 30 runs per label, they’re too uncertain to call either way. Three models ignoring the label doesn’t mean models are outgrowing SemVer. Those three did, and the next one you switch to might not. We also didn’t test whether another versioning scheme would work better for agents.

What to do with your version numbers

If you maintain a library, keep using SemVer, and keep it honest. Agents recognize it, and some of them act on it. A breaking change shipped as a patch is exactly the case where those agents skip the check, and every broken main in our runs came with a patch or a minor label. Your version numbers are part of the experience agents have with your product, the same way your docs and CLI help are.

If agents upgrade your dependencies, don’t let the label decide how much checking happens. Some models check every update on their own, and some check only when the version number looks risky. Build verification into the workflow, for example with CI that must pass on every dependency branch, so a patch gets the same checks as a major release.

If you evaluate agents, add the version number to your list of hidden variables. With Claude Haiku 5.5 or Sonnet 5.5, a scenario that bumps a patch version measures a different agent than one that bumps a major version. Test with the models your developers actually use, because testing only Claude Opus 5.5 would have told you agents ignore the label, and testing only Claude Haiku 5.5 would have told you the opposite. Read the trajectories too, because a run where the agent bumped a patch and got lucky passes just like a run where it checked.

So, do AI coding agents care about SemVer?

Some do. We expected agents to read the code and ignore the label, and Claude Opus 5.5, GPT-6.1 Sol and GPT-6.1 Terra did exactly that. Others used the label to decide how hard to look, the same way a developer skimming a Dependabot PR would.

A version number can’t prove that an upgrade is safe, for agents or for people. The version number tells an agent how worried to be. Only the tests tell it whether it should be.

The post Do AI coding agents care about SemVer? appeared first on Microsoft for Developers.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 Microsoft DevBlogs(Do AI coding agents care about SemVer?),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗