Together AI 发布 Together Compiler:智能体让推理最高提速 2.6 倍

Together AI 推出了 Together Compiler(TKCC),这是一个由智能体驱动的系统,可随着生产流量变化持续重新优化推理。Together Compiler 已在 Together 的多个无服务器端点上运行。它对开发者的核心主张是:一次性调优的引擎会过时,因此调优不应停止。
Together Compiler 是 Together AI 的持续编译循环,使用编码智能体重写模型的前向传播。它接收服务器启动命令、流量画像和硬件配置。智能体提出更快的前向传播方案,并由一个常驻引擎逐一测量。胜出的代码被部署到服务引擎中。当流量发生漂移时,循环会再次运行。
Together AI 团队报告称,其流量形态从不会静止不变。某个国家的提示长度约为 97K Token,而其他地区为 70K。一位客户的 P90 输入长度在 3 天内从 18K Token 增长到 33K Token。
在随机数据上调优的内核也可能产生误导。随机输入激活的 MoE 专家数量比真实流量多 13% 到 38%。一个由智能体编写的 top-k 内核在随机数据上耗时 20 µs,而在 70K 上下文下的真实数据上,它耗时最高达 62 ms。
大多数智能体循环会为每次实验重启 vLLM、SGLang 或 TensorRT-LLM。研究团队测得,MiniMax M3.1 Flash 在 TensorRT-LLM 上的冷启动时间为 761 秒。Together Compiler 只启动一次引擎。智能体通过 RPC 提交候选前向传播。服务器替换前向函数指针并重新捕获 CUDA 图。
两个测量层级在速度与保真度之间取得平衡。快速层级处理 12 层,用于快速筛选。慢速层级处理 60 层,用于在真实分布上确认收益。两个层级测试的候选方案数量是单一慢速层级的 8.1 倍(1,354 对 168)。它们还找到了一个快 7.2% 的解决方案。Together AI 团队报告称,其智能体实验吞吐比引擎重启基线高 252 倍。
成本模型在智能体启动之前,对张量并行、专家并行、数据并行和上下文并行的布局进行排序。它融合了规格表数据、微基准测试和实测前向传播结果,还利用了实测的 MoE 专家不均衡和请求长度不均。在 Kimi K3 上,它找到了一种生产布局,TPS 最高提升 24%。
由专家整理的知识库在 12 小时后带来的加速反而低了 25% 到 31%。被要求把前向传播做到“尽可能快”的朴素智能体,表现超过了专家策略。10 次试验后,集成方法的得分为 6.2 倍对 4.9 倍。自定义日志结构占用了多达 43% 的工具调用。Together AI 现在直接向智能体提供原始引擎源代码,而非中间表示。
在 SemiAnalysis InferenceX 的 AgentX 工作负载上,Together Compiler 相比最佳开放配置的 TPS 最高领先 2.6 倍。每组配置在最多 2 个节点上运行不超过 24 小时。
批量大小为 1 时,每用户 P90 TPS(Together Compiler 对比 InferenceX 最佳结果,自我报告):
对于 MiniMax,成本模型选择了 TP8,而非基线的 TP4。在受内存带宽限制的解码过程中,这使每块 GPU 的权重传输量减半。
在批量大小为 1 的 Llama 1B 上,它相比 Hazy Research 的 megakernels、Mirage 和 CUDA-graph 基线最高领先 1.7 倍。除 LLM 之外,它还将 8 个科学模型加速了 1.5 倍到 6.4 倍,且输出位精确一致。使用 /goal 的 Claude Code 达到 1.2 倍到 4.4 倍,但在 AIFS-ENS 2.0 上失败。
需要与我们合作,推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等?欢迎联系我们。
《Together AI 发布 Together Compiler:智能体让推理速度最高提升 2.6 倍》一文首发于 MarkTechPost。
译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster)
Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop.
TL;DR
- System: It is an optimization system, tested on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B.
- Runs on: NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2. Each run used at most 2 nodes per model.
- Performance: Up to 2.6x higher TPS than the best published InferenceX configurations, in under 24 hours per setting.
- Best: 1,111 P90 TPS per user on MiniMax M3.1 Flash (B200, batch size 1), 1.8x over the 633 TPS baseline.
- Worst: GLM 5.3 on MI355X moved only from 463 to 480 TPS (about 4%).
What is Together Compiler?
Together Compiler is a continual compilation loop from Together AI that uses coding agents to rewrite a model’s forward pass. It takes a server launch command, a traffic profile and a hardware configuration. Agents propose faster forward passes, and a persistent engine measures each one. The winning code ships into the serving engine. When traffic drifts, the loop runs again.
Why does inference need continual compilation?
Together AI team reports that its traffic shape never sits still. Prompt lengths in one country ran about 97K tokens versus 70K elsewhere. One customer’s P90 input length grew from 18K to 33K tokens in 3 days.
Kernels tuned on random data can also mislead. Random inputs activated 13 to 38% more MoE experts than real traffic. One agent-written top-k kernel took 20 µs on random data. On real data at 70K context, it took up to 62 ms.
How does the persistent engine work?
Most agent loops restart vLLM, SGLang or TensorRT-LLM for every experiment. The research team measures 761 seconds of cold startup for MiniMax M3.1 Flash on TensorRT-LLM. Together Compiler starts the engine once. Agents submit a candidate forward pass over RPC. The server swaps the forward function pointer and re-captures CUDA graphs.
Two measurement tiers balance speed and fidelity. A fast tier serves 12 layers for quick screening. A slow tier serves 60 layers to confirm gains on real distributions. Two tiers tested 8.1x more candidates than one slow tier (1,354 vs 168). They also found a solution 7.2% faster. Together AI team reports 252x higher agent experiment throughput than an engine-restart baseline.
How does it choose a parallelism layout?
A cost model ranks tensor, expert, data and context parallel layouts before agents start. It mixes spec-sheet numbers, microbenchmarks and measured forward passes. It also uses measured MoE expert imbalance and uneven request lengths. On Kimi K3, it found a production layout with up to 24% higher TPS.
What did agents need from human experts?
An expert-curated knowledge base produced 25 to 31% lower speedup after 12 hours. Naive agents told to make the forward pass “as fast as possible” beat expert strategies. The ensemble scores were 6.2x versus 4.9x after 10 trials. Custom logging structures ate up to 43% of tool calls. Together AI now gives agents raw engine source code, not an intermediate representation.
Benchmarks: how fast is Together Compiler?
On SemiAnalysis InferenceX‘s AgentX workload, Together Compiler beat the best open configurations by up to 2.6x TPS. Each setting ran under 24 hours on at most 2 nodes.
Batch size 1, P90 TPS per user (Together Compiler vs InferenceX best, self-reported):
- MiniMax M3.1 Flash (B200): 1,111 vs 633
- DeepSeek v4.1 Flash (GB300): 1,205 vs 943
- GLM 5.3 (B200): 417 vs 329
- Kimi K3 (GB300): 450 vs 423
- GLM 5.3 (MI355X): 480 vs 463
For MiniMax, the cost model picked TP8 over the baseline’s TP4. That halves each GPU’s weight traffic in memory-bound decode.
On Llama 1B at batch size 1, it beat Hazy Research megakernels, Mirage and CUDA-graph baselines by up to 1.7x. Beyond LLMs, it sped up 8 scientific models by 1.5x to 6.4x with bit-exact outputs. Claude Code with /goal reached 1.2x to 4.4x and failed on AIFS-ENS 2.0.
Auto-play</button></div> </div> <div class="panel" data-p="1"> <div class="legend">Time before a candidate can be measured (MiniMax M3.1 Flash, TensorRT-LLM, paper Table 1)</div> <div class="row"><span class="lb">Cold engine start (status quo)</span><div class="track"><div class="bar b0" data-w="100"></div></div><span class="vv">761 s</span></div> <div class="row"><span class="lb">Hot reload, slow tier (60 layers)</span><div class="track"><div class="bar b2" data-w="5.9"></div></div><span class="vv">45 s</span></div> <div class="row"><span class="lb">Hot reload, fast tier (12 layers)</span><div class="track"><div class="bar b1" data-w="2.2"></div></div><span class="vv">17 s</span></div> <div class="grid2" style="margin-top:12px"> <div class="stat"><div class="v">252x</div><div class="k">more agent experiments than an engine-restart baseline</div></div> <div class="stat"><div class="v">8.1x</div><div class="k">more candidates with 2 tiers vs 1 slow tier (1,354 vs 168)</div></div> <div class="stat"><div class="v">7.2%</div><div class="k">faster final solution from the two-tier setup</div></div> <div class="stat"><div class="v">11x</div><div class="k">lower idle time per agent</div></div> </div> </div> <div class="panel" data-p="2"> <div class="sel"><button class="tab on" data-m="p90">P90 TPS per user, batch size 1</button><button class="tab" data-m="llama">Llama 1B, batch size 1 (B200)</button></div> <div class="legend" id="tk-lg"></div> <div id="tk-bars"></div> <p class="note">Self-reported by Together AI. InferenceX baselines as of Oct 27, 2026. Under 24 hours, at most 2 nodes per model.</p> </div> <div class="panel" data-p="3"> <div class="grid2"> <div class="card"><h4>Knowledge base <span>+</span></h4><div class="more">An expert-curated knowledge base gave <span class="big">25 to 31%</span> lower speedup after 12 hours. Agents wasted time searching it.</div></div> <div class="card"><h4>Expert strategies <span>+</span></h4><div class="more">Naive agents ("make it as fast as possible") beat expert-assigned strategies: <span class="big">6.2x vs 4.9x</span> after 10 trials.</div></div> <div class="card"><h4>Custom logging <span>+</span></h4><div class="more">Human-designed research logs consumed up to <span class="big">43%</span> of tool calls per trial.</div></div> <div class="card"><h4>No intermediate representation <span>+</span></h4><div class="more">Raw engine source let agents rewrite Kimi K3's MLA metadata kernel for a <span class="big">1.7 to 2.5x</span> kernel speedup.</div></div> <div class="card"><h4>Megakernels on GPUs <span>+</span></h4><div class="more">Rarely paid off: only <span class="big">0.22%</span> faster on AMD GLM 5.3. Trainium2 was the exception.</div></div> <div class="card"><h4>Still the bottleneck <span>+</span></h4><div class="more">Agents spend over <span class="big">50%</span> of their time waiting on measurement gates.</div></div> </div> </div> <div class="foot"><span>Sources: Together AI paper and blog</span><span class="mtp">© Marktechpost</span></div> </div> <script> (function(){ var R=document.getElementById('tkcc-x'); function rs(){try{parent.postMessage({tkccH:R.offsetHeight+40},'*')}catch(e){}} var steps=[ 'Together reports traffic keeps changing. One customer\'s P90 input length grew from <em>18K to 33K</em> tokens in 3 days. Random test data activates <em>13 to 38%</em> more MoE experts than real traffic.', 'A cost model ranks parallelism layouts before agents start. It blends spec sheets, microbenchmarks and measured passes. On Kimi K3 it found a layout with up to <em>24%</em> higher TPS.', 'Multiple coding agents work in parallel and share findings on a message board. High-frequency communication made search <em>4.7x</em> faster in ablations.', 'The engine starts once. Each candidate is hot-reloaded: swap the forward pointer, re-capture CUDA graphs. A <em>12-layer</em> fast tier screens; a <em>60-layer</em> slow tier confirms.', 'After the budget runs out, verification tests check correctness and KV cache safety. The best config and forward pass go to production. Result: up to <em>2.6x</em> TPS over InferenceX best.']; var cur=0,auto=null,nodes=R.querySelectorAll('.node'); function show(i){cur=(i+5)%5;nodes.forEach(function(n,k){n.classList.toggle('on',k===cur)});R.querySelector('#tk-d').innerHTML=steps[cur];R.querySelector('#tk-pg').style.width=((cur+1)*20)+'%';} nodes.forEach(function(n,k){n.onclick=function(){show(k)}}); R.querySelector('#tk-next').onclick=function(){show(cur+1)}; R.querySelector('#tk-prev').onclick=function(){show(cur-1)}; R.querySelector('#tk-auto').onclick=function(){var b=this;if(auto){clearInterval(auto);auto=null;b.textContent='
Auto-play';b.classList.remove('pulse')}else{auto=setInterval(function(){show(cur+1)},2600);b.textContent='❚❚ Pause';b.classList.add('pulse')}};
show(0);
var data={p90:{lg:'<i style="background:linear-gradient(90deg,#5b7cff,#ff4f8b)"></i>Together Compiler<i style="background:#3a3d6e"></i>InferenceX best',max:1300,rows:[['MiniMax M3.1 Flash · B200',1111,633],['DeepSeek v4.1 Flash · GB300',1205,943],['DeepSeek v4.1 Flash · B200',813,671],['Kimi K3 · GB300',450,423],['GLM 5.3 · MI355X',480,463],['GLM 5.3 · B200',417,329]]},
llama:{lg:'<i style="background:linear-gradient(90deg,#5b7cff,#ff4f8b)"></i>Together Compiler<i style="background:#3a3d6e"></i>Best baseline per length (megakernel or CUDA graph, output tokens/s)',max:2300,rows:[['OSL 256',2118,1649],['OSL 1,024',2011,1569],['OSL 4,096',1945,1414],['OSL 16,384',1871,1123],['OSL 65,536',1482,892]]}};
function bars(m){var d=data[m],h='';d.rows.forEach(function(r){h+='<div class="row"><span class="lb">'+r[0]+'</span><div style="flex:1"><div class="track" style="height:20px;margin-bottom:3px"><div class="bar b1" data-w="'+(r[1]/d.max*100)+'">'+r[1].toLocaleString()+'</div></div><div class="track" style="height:20px"><div class="bar b0" data-w="'+(r[2]/d.max*100)+'">'+r[2].toLocaleString()+'</div></div></div></div>'});
R.querySelector('#tk-lg').innerHTML=d.lg;R.querySelector('#tk-bars').innerHTML=h;grow(R.querySelector('#tk-bars'));}
function grow(el){var b=el.querySelectorAll('.bar');b.forEach(function(x){x.style.width='0'});setTimeout(function(){b.forEach(function(x){x.style.width=(x.textContent?Math.max(x.dataset.w,9):Math.max(x.dataset.w,1.5))+'%'})},60);setTimeout(rs,1200);}
R.querySelectorAll('[data-m]').forEach(function(t){t.onclick=function(){R.querySelectorAll('[data-m]').forEach(function(z){z.classList.remove('on')});t.classList.add('on');bars(t.dataset.m)}});
R.querySelectorAll('.tabs .tab').forEach(function(t){t.onclick=function(){var i=t.dataset.t;R.querySelectorAll('.tabs .tab').forEach(function(z){z.classList.toggle('on',z===t)});R.querySelectorAll('.panel').forEach(function(p){p.classList.toggle('on',p.dataset.p===i)});if(i==='1')grow(R.querySelector('[data-p="1"]'));if(i==='2')bars(R.querySelector('[data-m].on').dataset.m);rs();}});
R.querySelectorAll('.card').forEach(function(c){c.onclick=function(){c.classList.toggle('open');c.querySelector('h4 span').textContent=c.classList.contains('open')?'−':'+';setTimeout(rs,400)}});
window.addEventListener('load',rs);window.addEventListener('resize',rs);setTimeout(rs,300);
})();
</script>
</body></html>
" style="width:100%;max-width:880px;height:600px;border:0;display:block;margin:0 auto;" loading="lazy" title="Together Compiler interactive explainer">
How does it compare with other agentic optimization systems?
| Metrics | Together Compiler | Cursor + NVIDIA multi-agent | AMD GEAK | Apple RSI-Inference Bench |
|---|---|---|---|---|
| Type | Production optimization loop | Multi-agent kernel optimizer | Triton kernel agent framework | Benchmark for coding agents |
| Scope | Full forward pass inside the serving engine | 235 standalone CUDA kernels (SOL-ExecBench) | Individual Triton kernels | 37 tasks editing vLLM or SGLang |
| Measurement | Persistent engine, hot reload, 2 tiers, real traffic | SOL-ExecBench on 27 B200 GPUs | TritonBench-revised and ROCm benchmark | Verifier builds the engine and serves the model |
| Hardware | B200, B300, GB300, H100, MI355X, Trainium2 | B200 | AMD Instinct MI300X | H100, B200 |
| Main result | Up to 2.6x TPS vs InferenceX best | 1.38x geomean, wins on 149 of 235 | Up to 2.59x speedup, up to 63.33% correctness | Scores agents; no single system result |
| Time budget | Under 24 hours per setting | 3 weeks | Not disclosed | Up to 8 hours per task |
| Availability | Limited beta for Together customers | Results in public repo | Open source | Apple Sample Code License |
Key Takeaways
- Together Compiler re-optimizes inference continually as traffic shifts.
- Persistent engines with hot reload deliver 252x more agent experiments.
- Up to 2.6x TPS over the best InferenceX configs within 24 hours.
- Naive agents beat expert knowledge bases and curated strategies.
- Self-reported; gains range from about 4% to 2.6x across settings.
Check out the paper, the Together Compiler blog and the Polymorphic Inference post. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster appeared first on MarkTechPost.
正文保留英文原文,标题与摘要由机器翻译。
原文发布于 MarkTechPost(Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。