让开放模型胜任长周期智能体工作负载

作者:Parth Sareen 和 Zain Hasan(Together AI)
编码智能体的流量模式有些特别。智能体每走一步,运行框架(harness)都会把到目前为止的整段对话发给模型:系统提示词、每一条工具定义、你的请求、智能体读过的每个文件,以及它执行过的每条命令。而返回的内容通常只是一个几十 Token 长的工具调用。然后这件事会再来一遍,带着上次的全部内容,再加一点点新的。
这种“输入序列长度(ISL)大、输出序列长度(OSL)小”的特征,是大多数智能体式工作负载的共性。如果 GPU 每一轮都要从头处理整段对话记录,那么长会话会随着每一步变得越来越慢、越来越贵。前缀缓存挽救了其中大部分计算。问题在于,稍有不慎,前缀缓存也非常容易被破坏。
我们是从两个相反的方向切入这个问题的。Ollama 做的是运行框架侧的集成,也就是决定每次请求里放什么内容;Together AI 做的是推理引擎,也就是决定请求到达后如何处理。我们不断发现,双方各自的大量工作,其实都是在保护缓存不被另一方的行为破坏,所以我们想把这些内容集中写下来,帮助智能体开发者和基础设施开发者。
在 GPU 眼中,一次智能体回合是什么样的
每个请求都要经过两个阶段。第一个是预填充(prefill):引擎在一次大规模并行处理中读取整个提示词,并生成第一个 Token。GPU 擅长这件事,因为大量数学运算是一次性完成的,从而最大化了算术强度。预填充决定了在出现任何新 Token 之前你需要等待多久。第二个是解码(decode):模型逐个 Token 写出答案的其余部分,而每一个新 Token 都需要回读并对其之前的全部内容做注意力计算。解码受限于内存带宽多于受限于算力,它决定了文本流出的速度。
一次普通的聊天回复会在两个阶段都花时间,而智能体的一轮交互则以预填充为主,因为输入是整个会话,而输出通常只是一次简短的工具调用。推理模型还会在此基础上加上思考 Token,有时数量还很多,但当会话增长到数万 Token 时,输入端仍然占主导。每一轮的输入都变得更长,而输出大致保持同样的规模——几次工具调用。
所以对智能体来说,「这个模型有多快?」大体上意味着「它能多快地读完此前各轮交互的冗长历史记录?」要快速处理带有大量历史轮次的任务,办法就是把它缓存起来直接读回,而不是重新计算。
理解 KV 缓存(其实前缀缓存也一样)最直观的方式,就是把它类比成阅读。想象一下,为了读一页书上的每一个新词,你都得从最上面重新开始。你读得越往下,速度就越慢,因为你不得不一直重读已经读过的内容。这就是没有缓存的模型。而缓存能在单个请求内解决这个问题:引擎保留它此前对各个 Token 所做的工作(以中间的 key、value 对形式存储),只计算新的那一个。
前缀缓存在多个请求之间做的是同一件事。智能体会话的第 7 轮,就是第 6 轮再加上一点新内容,所以如果引擎仍然保留着第 6 轮的计算成果,它就只需要处理新增的那部分。
这笔账算起来很简单。如果每一轮新增的量相同,那么不做缓存的十轮要处理 1 + 2 + … + 10 = 55 个单位的提示词。有了缓存,它们只处理 10 个单位。到了一百轮——对编码任务来说并不罕见——就是 5,050 对 100。真实的每轮规模各不相同,所以把这些数字当作理想化的情况来看。带缓存的会话也并非完全免费,因为每个新 Token 仍然必须关注缓存中的那些 Token。但规律是成立的:不做缓存时,你重复做的功会随会话长度的平方增长。
问题在于,前缀缓存从提示词的开头开始匹配,一直到第一个不同的 Token 就停下。第一个差异点之后的所有内容都必须重新处理。
在提示词末尾追加内容不产生额外开销。改动更靠前的任何东西,哪怕只改一个字符,都意味着其后的所有内容都要重新处理。如果改动靠近开头,你实际上等于扔掉了整个会话,现在必须重建缓存。本文其余几乎所有内容,都是这一条规则的推论。
harness 决定第一个改动落在哪里,引擎决定下一个请求到来时缓存是否还在。我们先从 harness 说起。
在 Ollama 这边,我们推出了通过 ollama launch 命令接入你惯用 harness 的能力。这样你就可以把 Claude Code、Codex、Pi(以及更多)这类 harness 与 DeepSeek-V4.1-Flash、GLM-5.3 等开放模型连接起来,同时保持你现有的工作流。
把 harness 接入 API 看起来可能很简单,但 harness 与模型之间存在许多层转换,双方的预期也各不相同。这往往是缓存失效(带来时间和金钱成本)、智能体行为无法解释,以及 harness 直接报错的原因。
如今 harness 携带的工具相当多。如果每个定义都提前写进提示词,你就会把上下文花在任务根本用不到的工具上,模型挑选正确工具的能力也会变差。较新的 harness 增加了工具搜索功能来缓解这一问题:智能体可以自行搜索到某个工具,然后加载并使用它。
假设你有一个具备该能力的个人助理智能体。如果你让这个智能体把你标星的事件导出到日历,它会搜索“calendar”,加载 save_calendar 定义,然后调用它。
对缓存来说,问题在于这个定义放在哪里。
最直接的做法是把该定义加入工具列表。这行得通,但工具列表位于提示词靠近顶部的位置,在对话之前。因此一旦前缀发生变化,它之后的所有内容都要重新处理。在长会话中,这可能是数十万个 Token。更好的做法是把定义放在末尾,也就是新内容本来就会去的地方。
只不过并非每个模型都支持这种做法。有些聊天模板并不预期工具定义出现在对话中途,调用会报错。因此,harness 能否在不付出代价的情况下延迟加载工具,取决于模型。用户不该需要知道这些,但任何构建 harness 的人都必须知道。
顶部那些会发生变化的内容
破坏缓存最简单的方式,就是在系统提示词里放入逐轮变化的内容,比如当前时间戳、git status,或者智能体不断切换的工作目录。如果运行框架每一轮都用新值重新渲染系统提示词,那么每一轮都会是一次完整的重新读取。
在一个会话内固定、但在不同会话之间会变化的内容,比如用户的名字或会话开始的日期,不会损害该会话本身。但它们会阻止新会话复用已经缓存好的配置,这一点我们会在引擎侧展开说。无论哪种情况,解决办法都一样:把这些信息一次性放进任务消息里,并让系统提示词在任何地方都保持一致。
这个问题我们是在 Ollama 侧遇到的。在 GLM-5.3-Flash 中,推理强度并不是由引擎在旁单独处理的一个独立设置。聊天模板会把它写进最前面的第一个系统区块,位于工具和对话之前。
假设你正深入执行一个长任务,而且进展不顺。你把推理强度从低调到高。这本身是合理的做法,但下一个请求与上一个请求几乎从第一个 Token 开始就不同了。这意味着整个会话都要重新处理。在长时间运行中,仅仅为了改一个设置,就可能花掉数十万个 Token。
我们的建议是,一开始就选好推理强度,然后保持不变。如果你确实需要更多思考,通常以更高的设置开启一个新会话,比继续旧会话更便宜,而且你还能得到一个干净的上下文。你也可以在修改推理强度之前先做压缩,以减少提示词 Token 的数量。
工具的输出会成为下一个请求的一部分,也会成为其后每一个请求的一部分。如果智能体本只需要 40 行,却读了一个 3000 行的文件,这笔代价你并不是只付一次;你会在之后每一个回合都为此付出(即使内容被缓存,你也白白占用了上下文窗口)。为应对这一点,智能体框架(harness)通常会裁剪工具输出。这样做是对的,但如何裁剪工具输出很关键。
下面是一种很容易在无意中造成的失败。任务是为日历导出功能修复一个时区 bug。智能体读取 calendar.py,而工具返回了 640 行文件中的前 100 行,却没有说明它在哪里停下了。真正关键的函数 convert_timezone 位于第 420 行。模型完全不知道自己漏掉了内容。
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Making open models work for long-horizon agentic workloads)
By Parth Sareen and Zain Hasan (Together AI)
Coding agents have an odd traffic pattern. Every time the agent takes a step, the harness sends the model the entire conversation so far: the system prompt, every tool definition, your request, every file the agent has read, and every command it has run. What comes back is usually a tool call a few dozen tokens long. Then it happens again, with everything from last time plus a little more.
This large input sequence length (ISL) and small output sequence length (OSL) is characteristic of most agentic workloads. If the GPU had to work through that whole transcript from scratch on every turn, long sessions would get slower and more expensive with every step. Prefix caching is what saves most of that work. The problem is that if you’re not careful it’s also very easy to break the prefix cache.
We come at this problem from opposite ends. Ollama works on harness integrations which means deciding what goes into each request. Together AI works on the inference engine which means deciding what happens when that request arrives. We kept finding that much of our work on each side is about protecting the cache from things the other side can do to it, so we wanted to write it all down in one place to help agent and infra builders alike!
What an agent turn looks like to a GPU
Every request goes through two phases. The first is prefill: the engine reads the whole prompt in one big parallel pass and produces the first token. GPUs are good at this because it’s a lot of math done all at once which maximizes arithmetic intensity. Prefill determines how long you wait before any new tokens appear. The second is decode: the model writes the rest of the answer one token at a time, and each new token requires reading back and attending through everything before it. Decode is limited by memory bandwidth more than by compute, and it determines how fast the text streams out.

A normal chat response spends time in both phases while an agent turn is mostly prefill, because the input is the whole session and the output is usually a short tool call. Reasoning models add thinking tokens on top of that, sometimes a lot of them, but as a session grows into the tens of thousands of tokens the input still dominates. Each turn the input gets longer, and the output stays roughly the same size — a few tool calls.
So for agents, “how fast is this model?” mostly means “how fast can it get through a long historical account of previous turns?” The way to process a task with lots of previous turns quickly is to just cache it and read it back in rather than recomputing.
The easiest way to picture the KV cache (or even the prefix cache, for that matter) is reading. Imagine that to read each new word on a page, you had to start again from the top. The further down the page you got, the slower you’d go, because you’d keep having to reread everything you’d already read. That’s a model with no cache. The cache solves this within a single request: the engine keeps the work it did on earlier tokens (stored as intermediate key, value pairs) and computes only the new one.
Prefix caching does the same thing across requests. Turn 7 of an agent session is turn 6 plus a bit more, so if the engine still holds turn 6’s work, it only has to process the new part.
The arithmetic is simple. If each turn adds the same amount, ten turns without caching process 1 + 2 + … + 10 = 55 units of prompt. With caching, they process 10. At a hundred turns, which isn’t unusual for a coding task, it’s 5,050 versus 100. Real turns vary in size, so treat these as idealized numbers. A cached session isn’t completely free either, since every new token still has to attend to the cached ones. But the pattern holds: without caching, the work you redo grows with the square of the session length.
The catch is that a prefix cache matches from the beginning of the prompt up to the first token that’s different, and stops there. Everything after that first difference has to be processed again.
Adding to the end of the prompt costs nothing extra. Changing anything earlier, even a single character, means re-processing everything after it. If the change is near the top, you’ve effectively thrown away the whole session and now must rebuild the cache. Almost everything else in this post follows from that one rule.
The harness decides where that first change lands. The engine decides whether the cache is still there when the next request shows up. We’ll start with the harness.
On the Ollama side, we shipped the ability to connect your favorite harness through the ollama launch command. So you can connect harnesses like Claude Code, Codex, Pi (and more) with open models like DeepSeek-V4.1-Flash and GLM-5.3 and maintain your existing workflows.
Connecting a harness to an API might seem easy, but there are many layers of translation and varying expectations between a harness and a model. This often is the cause of cache breakages (incurring time and monetary costs), unexplainable agent behavior, and the harness outright erroring.
Harnesses carry quite a few tools now. If every definition goes into the prompt up front, you spend context on tools the task will never use, and the model gets worse at picking the right one. Newer harnesses have added a feature for tool search to alleviate this, where the tool can be loaded and used by the agent itself after searching for it.
Let’s say you have a personal assistant as an agent which has this capability. If you ask the agent to export the events you’ve starred to a calendar, and it searches for “calendar,” loads the save_calendar definition, and calls it.
For the cache, the question is where that definition goes.
The obvious approach is to add the definition to the tool list. That works, but the tool list sits near the top of the prompt, before the conversation. So if the prefix changes everything after it gets processed again. On a long session, that can be hundreds of thousands of tokens. The better approach is to put the definition at the end, where new content goes anyway.
Except not every model supports that. Some chat templates don’t expect a tool definition halfway through a conversation, and the call errors out. So whether a harness can load tools late without paying for it depends on the model. Users shouldn’t have to know this, but anyone building a harness does.
Things at the top that change
The easiest way to wreck the cache is to put something in the system prompt that changes from turn to turn, like a current timestamp, the git status, or a working directory the agent keeps moving through. If the harness re-renders the system prompt with fresh values on every turn, every turn is a full re-read.
Things that are fixed for a session but differ between sessions, like the user’s name or the date the session started, don’t hurt that session. They do stop new sessions from reusing a setup that’s already cached, which we’ll get to on the engine side. Either way, the fix is the same: put that information in the task message, once, and keep the system prompt identical everywhere.
We hit this one on the Ollama side. In GLM-5.3-Flash, reasoning effort isn’t a separate setting the engine handles on the side. The chat template writes it into the very first system block, ahead of the tools and the conversation.
Let’s say you’re deep into a long task and it isn’t going well. You raise effort from low to high. That’s a reasonable thing to do, but the next request now differs from the last one almost at the first token. Which means the whole session gets processed again. On a long run, that can be hundreds of thousands of tokens spent on changing one setting.
Our advice is to pick an effort level at the start and leave it there. If you really need more thinking, it’s often cheaper to start a fresh session at the higher setting than to continue the old one, and you get a clean context out of it too. You could also compact before changing the reasoning value to decrease the amount of prompt tokens.
A tool’s output becomes part of the next request, and of every request after it. If the agent reads a 3,000-line file when it needed forty lines, you don’t pay for that once; you pay for it on every remaining turn (even if it is cached you lose out on the context window). To combat this, harnesses often cut tool output down. That’s the right call, but how you cut tool output matters.
Here’s a failure that’s easy to build by accident. The task is to fix a timezone bug in a calendar export. The agent reads calendar.py, and the tool returns the first 100 lines of a 640-line file without saying it stopped. The function that matters, convert_timezone, is at line 420. The model has no idea it’s missing anything.
The fix is a single line in the output, telling the model it got lines 1 to 100 of 640 and that it can search the file or request another range. With that, the model searches for “timezone,” finds line 420, and reads lines 410 to 450. Cutting output saves money. Cutting it without saying so gets you a model confidently editing code it never saw.
Open models like Kimi K3, GLM-5.3, and GLM-5.3-Flash are now pretty good at splitting a big task and handing pieces to subagents. How the harness starts each subagent decides how much of the parent’s cache it can reuse.
One option is for the subagent to inherit everything: the same system prompt, tools, and conversation, plus its assignment. Side chats work this way. They branch off the parent session, so the whole prefix is already cached. The other option is to send a brief: the same system prompt and tools, then only the files and instructions the subagent needs. In that case the cache covers the setup and nothing more.
Inheriting sounds strictly better, but it isn’t. A cached prefix is cheap to process, not free. Every token the subagent generates still has to read through it, and it occupies GPU memory the whole time. A reviewer that needs two files and a checklist probably doesn’t need three hundred turns of history. And if you give a subagent its own system prompt, it shares nothing with the parent at all, not even the setup.
You can do everything above perfectly and still not benefit. The cache has to still exist, and the next request has to reach the machine that holds it.
At Together AI, we think of an inference engine as a machine for keeping GPUs busy. A single GPU costs around $50k to rent per year, so ideally it should never sit idle, and at any moment it’s serving hundreds of users who are each at a different point in their request. Agent traffic pushes on those constraints in a few specific ways.
Routing requests for optimal cache hits
A session’s cache lives on a particular node in a particular data center. If the next turn in a conversation lands anywhere else, that node has to process all earlier turns from scratch, and now two nodes are storing the same prefix. In order to route correctly we first need to pick a data center that runs the model, has capacity, and is where the session has been running. Then we need to pick a node inside it that has the previous turns cache.
There are two main ways to pick the node. The simple one is to use a key: the client sends something like prompt_cache_key with every request and then ideally we route every request with the same key to the same node. The other way is to hash the prefix and send the request to whichever node has the longest matching prefix cached, even if another session put it there.
That second option is the other reason to keep the system prompt identical. If every session of a harness starts with exactly the same system prompt and tool definitions, a brand-new session might be able to find that setup already cached. If you put unique identifiers in there then every session is guaranteed to start cold.
It also isn’t as simple as always sending the request to the warm node. Sometimes the warm node is overloaded with traffic, and then the choice is between waiting in line there or taking an idle node and paying to process the prefix again. In this load-aware routing paradigm sometimes the cache miss might be the better deal. So even a perfectly stable prompt doesn’t guarantee a hit if the inference engine on the backend is not implemented correctly.
Keeping the cache alive between turns
Routing only helps if there’s still a cache to route to. An agent will often make a tool call and then wait a couple of minutes while tests run. Keeping that session’s cache in GPU memory the whole time takes space active users need, and throwing it away means re-reading everything when the agent comes back from its tool calls. So we can also offload the cache to CPU memory while the session is idle and restore it for the next turn.
Memory is the constraint behind most of this. We can’t know ahead of time whether a response will be 30 tokens or 30,000, so instead of reserving room for the worst case max context length scenarios, we hand out cache memory in small blocks as each request needs them (PagedAttention), the way an operating system handles RAM. And we rebuild the running batch on every pass (continuous batching), so when an agent fires off five subagents at once, each new request takes a slot the moment one frees up instead of waiting for the slowest one in a batch to finish.
The upshot for anyone building a harness is that headroom for cache memory, not compute, is usually what limits how many requests a GPU can serve at once. Every oversized tool result, and every subagent carrying irrelevant history takes memory that another session’s cache could have used.
Agents also push the prefill/decode split to its extreme, with enormous prompts and tiny answers. To better optimize this we run the two phases on separate clusters. One is tuned for compute and does prefill; the other is tuned for memory bandwidth and does decode for hundreds of streams at once, with the cache shipped between them over RDMA. That way, a wave of long agent prompts doesn’t slow token streaming for everyone else.
Putting it all together: One session, both sides
Here’s what all of this looks like in a single session.
On the harness side, the agent finds the calendar tool and adds it at the end of the conversation, so nothing above it changes. It reads calendar.py, sees that it got only lines 1 to 100 of 640, searches for “timezone,” and reads the right section. Effort stays where it started. A reviewer subagent inherits the session to check agenda.ics against sessions.json.
On the engine side, every turn goes back to the node that has the session cached. While the tests run, the cache waits in CPU memory. The reviewer’s request slots into the running batch, and each turn processes only what’s new.
None of that is visible to the person using the agent; they just see it stay fast at turn ninety. But it only works if both sides do their part. A harness can keep the prompt perfectly stable and still lose the cache if the engine sends turn 7 to the wrong machine. An engine can route perfectly and still lose it if the harness changes the first line every turn.
Read more about building agents and open models on Parth’s blog and the Together AI blog.
原文为英文,正文由自动抓取并保留原文。
原文发布于 Ollama Blog(Making open models work for long-horizon agentic workloads),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。