Together Compiler:面向 AI 推理的持续编译

推出 Together Post-Training:模型更优、运行更快、成本更低 →
推出 Together Link:在您已在使用的运行框架内运行开放模型 →
开源 AI 技术栈:模型、推理、路由器、运行框架与工具 →
GLM-5.3 与 GLM 5.3 Flash 在 DeepSWE 上的对比 →
⚡ 按需 B200 现已在 Together GPU 集群上线 →
以 API 形式提供的高性能推理
面向批处理工作负载的推理
带 SLA 的按 Token 计费容量
专用模型推理
自定义硬件上的推理
专用容器推理
自定义模型的推理
在自有运行框架中运行开放模型
随时间不断优化的推理
探索顶尖开源模型
大规模可靠 GPU 集群
前沿规模下的定制基础设施
为 AI 构建开发环境
安全存储模型权重与数据
定义你自己的 RL 与 SFT 循环
启动托管式 SFT 和 DPO 任务
微调顶级开源模型
面向生产级 AI 的系统研究
我们的全部研究出版物
Together AI 技术文档
我们的最新新闻与博客文章
我们的开源演示应用
实用实施指南
用开放模型构建更优方案
在你的测试框架中运行开放模型
浏览我们的活动日历
开源 ROI 计算器
以 API 形式提供的高性能推理
面向批处理工作负载的推理
带 SLA 的基于 Token 的容量
专属模型推理
在自定义硬件上运行推理
专属容器推理
面向自定义模型的推理
在你自己的运行框架中运行开放模型
随时间持续提升的推理
探索顶级开源模型
规模化部署的可靠 GPU 集群
前沿规模的自定义基础设施
为 AI 构建开发环境
安全存储模型权重与数据
定义你自己的 RL 与 SFT 循环
启动托管式 SFT 与 DPO 任务
微调顶级开源模型
面向生产环境 AI 的系统研究
我们的全部研究论文
Together AI 技术文档
我们的最新新闻与博客文章
我们的开源演示应用
实用实施指南
用开放模型构建更好的应用
在你的 harness 中运行开放模型
浏览我们的活动日历
开源 ROI 计算器
Together Compiler:面向 AI 推理的持续编译
AI 推理工作负载是动态的。存在数千种工况,涵盖新模型、硬件平台、客户、用例和流量形态。再靠人工逐一优化这些工况,已不再具备可扩展性。
架构高度多样:距 Transformer 问世已过去约 11 年。我们或许以为会出现更多整合收敛,但 OpenRouter 上排名前 10 的开源 LLM 无一采用相同架构!
工作负载高度多样:Together 每天为数千家客户服务数万亿 Token,每一家的流量形态都不同(批大小、序列长度、MoE 专家不均衡、注意力稀疏度、会话时长、领域)。
硬件高度多样:H100、B200、B300、NVL72、VR、Groq 等各自都需要不同的策略。
理论上,智能体非常适合解决这个问题。它们能够适应新的工作负载,并在可能的优化空间中进行搜索,而不必依赖那些困扰人类专家和传统机器学习编译器的固定规则或启发式方法。然而,如今的智能体式系统受制于实验速度,形成了瓶颈。
我们推出 Together Compiler(TKCC),这是首个真正持续运行的优化循环,利用高吞吐智能体面向多样化的推理工作负载进行优化。
在不到 24 小时内,TKCC 在前沿开源模型(Kimi K3、GLM 5.3、MiniMax M3.1-Flash、DeepSeek v4.1-Flash)上实现了最高 2.6 倍的 TPS 提升,在 SemiAnalysis 的 InferenceX 低延迟基准测试中刷新了最优成绩——MiniMax M3.1-Flash 达到 1111 TPS,DeepSeek v4.1-Flash 达到 1205 TPS,并在其主力场景 Llama 1B、batch size 1 下,比现有的开源 megakernel 快 1.7 倍。此外,TKCC 还能泛化到语言模型推理之外的领域(天气建模、PDE、时间序列和生物学)。这一性能提升得益于 TKCC 相比基线智能体式性能优化系统高出 252 倍的实验吞吐。如今,Together Compiler 已在服务数万亿 Token。
流量在不断变化
作为推理服务提供商,我们看到用户行为多种多样,这些行为受到地域模式、新模型发布和突发性的影响。
地域模式:世界各地的用户可能有不同的模型偏好。我们观察到,某国工作时段流量的提示词长度是另一国的 1.4 倍(97k vs 70k)。
模型发布:用户需要时间才能弄清如何把新发布的模型整合进自己的工作流。起初,他们往往选择基于对话的交互与评估。而发布仅两周后,随着用户转向智能体式工作负载,提示词长度会迅速增加 80%。
突发性:客户可能在一天中的任意时段发送突发请求。提示词长度可能突然翻倍,如果处理不当,缓存命中率也会下降。
显然,我们必须针对每一次新的部署做专门的优化,以适应用户行为的不断变化。
Together Compiler 概览
TKCC 以流量形态、模型和硬件作为输入。TKCC 使用智能体为推理引擎生成一系列优化。
图 1:随着流量漂移,Together Compiler 会重新编译,智能体、常驻引擎与集成体构成一个循环。
TKCC 的核心洞察如下:
1. TKCC 在智能体优化过程中最大化实验吞吐,并最小化 GPU 空闲时间
大多数现有的智能体式推理优化系统要求智能体启动和拆除 VLLM、SGLang 或 TRT-LLM 等引擎,以测试每个新实验。然而,引擎启动时间长达数十分钟,严重阻碍了实验吞吐(见下表)。相反,TKCC 提出了一种“持久化引擎”的新范式:引擎只需启动一次,就能热重载任何候选前向传播,无需任何重启。为了确保持久化引擎始终有新的优化可测试,TKCC 会创建多个智能体,持续生成想法,并通过留言板通信,以并行推进各自的工作。
RPC 调用 + Together Compiler 设置
表 1:引擎启动成本高昂
2. TKCC 平衡测量吞吐与测量保真度之间的权衡
现有的智能体优化系统要么反复重新启动推理引擎,因而实验吞吐低下;要么在推理引擎之外孤立地优化独立内核,因而测量保真度差。优化独立内核的系统通常使用随机输入数据来评估它们。然而,现代架构使用 MoE 和稀疏 k 注意力等依赖数据的原语,输入数据分布会直接影响哪些优化策略可能胜出(见下图)。高保真输入对于有效优化很重要。TKCC 使用分层持久化引擎来平衡吞吐与保真度之间的权衡:“fast engines”加载少量层,适合快速迭代和低方差计时;而“slower engines”加载许多层,适合测试优化能否在真实数据分布上成立。
图 2:输入数据会改变优化策略。
图 3:Together Compiler 的两层架构优于其他方案。
3. TKCC 利用自定义成本模型为智能体选择并行策略(如 TP、EP、DCP 等),以实现最高性能。
现有系统已有成熟的方法,可在满足 SLA 要求的前提下选择推理配置。然而,选择理想配置面临同样的吞吐与保真度权衡:逐一测量每种并行配置是不现实的,尤其是在软件实现尚不成熟的情况下。一种朴素的做法是依据硬件的原始规格或微基准测试来选择并行策略。但这种方式无法充分刻画依赖数据的流量特征,例如 MoE 专家负载不均衡或 prompt 长度分布不均。我们设计了一个成本模型,采用分层测量策略——结合硬件规格、微基准测试和实测前向传播——来构建高保真的表征,同时快速剪枝配置搜索空间,以支持持续的流量漂移。
图 4:实测布局优于硬件规格所选布局,且最佳布局随并发变化而改变。
我们将 TKCC 与三类基线进行对比评估:手工调优的 MegaKernel、InferenceX 的引擎配置,以及来自气象建模、PDE、时间序列和生物模型等不同领域的流行 AI 代码库。
我们首先评估经典的 MegaKernel 工作负载:批大小为 1 的 Llama 1B。该模型架构已被人专家和传统机器学习编译器充分优化。TKCC 实现了 1.7 倍的 TPS 提升,并且随着序列长度增加仍能保持性能优势。
图 5:Together Compiler 在 H100 和 B200 上均优于 MegaKernel 和 CUDA graph 基线。
InferenceX AgentX 工作负载
原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。
查看英文原文(Together Compiler: Continual Compilation for AI inference)
Introducing Together Post-Training: Better models, faster runs, lower costs →
Introducing Together Link: Run open models inside the harness you already use →
The Open Source AI Stack: models, inference, routers, harnesses & tools →
GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →
⚡ On-demand B200s now available on Together GPU Clusters →
High-performance inference as APIs
Inference for batch workloads
Token-based capacity with SLAs
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
Run open models in your harness
Inference that gets better over time
Explore Top Open-Source Models
Reliable GPU clusters at scale
Custom infrastructure at frontier scale
Build development environments for AI
Store model weights & data securely
Define your own RL and SFT loop
Launch managed SFT and DPO jobs
Fine-tune top open-source models
Systems research for production AI
All our research publications
Technical docs for Together AI
Our latest news & blog posts
Our open-source demo apps
Practical implementation guides
Build better with open models
Run open models in your harness
Explore our events calendar
Open-source ROI calculator
High-performance inference as APIs
Inference for batch workloads
Token-based capacity with SLAs
Dedicated Model Inference
Inference on custom hardware
Dedicated Container Inference
Inference for custom models
Run open models in your harness
Inference that gets better over time
Explore Top Open-Source Models
Reliable GPU clusters at scale
Custom infrastructure at frontier scale
Build development environments for AI
Store model weights & data securely
Define your own RL and SFT loop
Launch managed SFT and DPO jobs
Fine-tune top open-source models
Systems research for production AI
All our research publications
Technical docs for Together AI
Our latest news & blog posts
Our open-source demo apps
Practical implementation guides
Build better with open models
Run open models in your harness
Explore our events calendar
Open-source ROI calculator
Together Compiler: Continual Compilation for AI inference
AI inference workloads are dynamic. There are thousands of regimes spanning new models, hardware platforms, customers, use cases, and traffic shapes. It’s no longer scalable to optimize each of these regimes by hand.
Architectures are highly variable: it’s been ~11 years since the Transformer. We might have thought there would be more consolidation, but none of the top-10 open-source LLMs on OpenRouter use the same architecture!
Workloads are highly variable: Together serves many trillions of tokens per day across thousands of customers, each with a different traffic shape (batch size, sequence length, MoE expert imbalance, attention sparsity, session duration, domain).
Hardware is highly variable: H100s, B200s, B300s, NVL72, VR, Groq, and more each require distinct strategies.
Agents are a great fit for this problem, in theory. They can adapt to new workloads and search over the space of possible optimizations without relying on fixed rules or heuristics that plague human experts and traditional machine learning compilers. However, today’s agentic systems are bottlenecked by the speed of experimentation.
We introduce Together Compiler (TKCC), the first trulycontinual optimization loop that leverages high-throughput agents to optimize to diverse inference workloads
In less than 24 hours, TKCC achieves up to 2.6x higher TPS across frontier open source models (Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash), sets the state-of-the-art on SemiAnalysis’ InferenceX benchmark for low-latency with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash, and outperforms available open-source megakernels by 1.7x on their workhorse Llama 1B batch size 1 setting. Furthermore, TKCC generalizes to domains beyond language model inference (weather modeling, PDE, time-series, and biology). This performance boost is enabled by TKCC’s 252x higher experiment throughput compared to baseline agentic performance optimization systems. Together Compiler is serving trillions of tokens today.
Traffic is continually changing
As an inference provider, we see a variety of user behaviors based on regional patterns, new model releases, and burstiness.
Regional patterns: Users across the world may have different model preferences. We’ve observed traffic for one country’s working hours to have 1.4x longer prompts than another country (97k vs 70k).
Model releases: Users take time to figure out how to integrate new model releases into their workflows. First, they tend to opt for chat-based interaction and evaluation. This is swiftly followed by an 80% increase in prompt length just two weeks after the release as users shift towards agentic workloads.
Burstiness: Customers may send bursts of requests at random times throughout the day. Prompt lengths can suddenly double and cache hit rates can fall if not handled carefully.
It’s clear that we must adapt to ever-changing user behavior with specialized optimizations for each new deployment.
Overview of Together Compiler
TKCC takes as input the traffic shape, model, and hardware. TKCC uses agents to generate a series of optimizations for an inference engine.
Figure 1: Together Compiler recompiles as traffic drifts, with agents, a persistent engine and an ensemble in a loop.
The core insights in TKCC are as follows:
1. TKCC maximizes experimentation throughput and minimizes GPU idle time during agent optimization
Most existing agentic inference optimization systems require agents to launch and teardown engines like VLLM, SGLang, or TRT-LLM to test each new experiment. However, engine start times are tens of minutes, significantly impeding experimentation throughput (see table below). Instead, TKCC proposes a new paradigm of “persistent engines” where the engine starts up once and can hot-reload any candidate forward pass, without any restarts. To ensure that the persistent engines constantly have new optimizations to test, TKCC spawns multiple agents that continually generate ideas and communicate via a message board to parallelize their efforts.
RPC call + Together Compiler setup
Table 1: Engine startup is expensive
2. TKCC balances the tradeoff between measurement throughput and measurement fidelity
Existing agent optimization systems either iteratively relaunch the inference engine, suffering from low experiment throughput, or optimize standalone kernels in isolation, outside of the inference engine, suffering from poor measurement fidelity. Systems that optimize standalone kernels often evaluate them using random input data. However, with modern architectures using data-dependent primitives like MoE and sparse-k attention, input data distributions have a direct effect on which optimization strategies may win (see figure below). High-fidelity inputs are important for effective optimization. TKCC uses a tier of persistent engines to balance the throughput-fidelity tradeoff: “fast engines” load a few layers and are ideal for rapid iteration and low-variance timings, while “slower engines” load many layers and are ideal for testing whether optimizations hold-up on real data distributions.
Figure 2: Input data changes optimization strategies.
Figure 3: Together Compiler’s two-tiered architecture outperforms alternatives.
3. TKCC utilizes a custom cost model to select parallelism strategies (e.g., TP, EP, DCP, etc.) for agents to achieve the highest performance
Existing systems have sophisticated methods for selecting inference configurations under SLA requirements. However, selecting the ideal configuration faces the same throughput-fidelity tradeoff: it is intractable to measure each parallelism configuration, especially because software implementations are immature. A naive approach is to select a parallelism strategy based on raw hardware specs or micro-benchmarks. However, this insufficiently captures data dependent traffic properties like MoE expert imbalance or uneven prompt length distributions. We designed a cost model that uses a tiered measurement strategy involving hardware specs, microbenchmarks, and measured forward passes to construct a high-fidelity representation while rapidly pruning the configuration search space to support continual traffic drift.
Figure 4: Measured layouts beat the hardware spec’s pick, and the best layout shifts with concurrency.
We evaluate TKCC against three different types of baselines: Hand-tuned megakernels, InferenceX engine configurations, and popular AI codebases from diverse domains like weather modeling, PDE, time series and biology models.
We start by evaluating the canonical MegaKernel workload: Llama 1B at batch size 1. This model architecture has been well-optimized by both human experts and traditional machine learning compilers. TKCC generates 1.7x higher TPS and continues to perform as sequence length increases.
Figure 5: Together Compiler outperforms megakernel and CUDA graph baselines on both H100 and B200
InferenceX AgentX workloads
We evaluate TKCC on the AgentX benchmark from SemiAnalysis. We start from the best-reported open source engine configurations for frontier models across inference engines and hardware platforms. TKCC consistently sets the performance frontier across all settings, outperforming highly-optimized baselines by up to 2.6x higher TPS.1
We also evaluate specifically on the low-latency use case at batch size 1 and set the state-of-the-art performance with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash. For these results, we use our cost model to select the parallelism configuration instead of using the InferenceX configuration. We observe speedups with this approach.
Figure 6: Together Compiler serves more tokens per GPU at the same speed per user on most models.
Figure 7: Together Compiler is faster per user than the best InferenceX configuration on all six model and chip pairs
We select popular open-source code bases across weather modeling, PDE, time-series, and biology. We show the geometric mean speedups of TKCC over simply using Claude Code with the /goal command, illustrating that the principles of TKCC generalize across domains. We have been using TKCC as a general purpose AI compiler beyond LLM inference.
Figure 8: Together Compiler finds larger speedups than Claude Code on three of four models
Polymorphic AI systems is the next frontier
For Together AI, the most exciting systems challenge has been Polymorphic Inference. Inference is no longer about optimizing for a single traffic shape for a single hardware platform and model, traffic is constantly and dynamically evolving, much like financial markets, and that dynamism has opened up a whole host of new research questions.
Check out our paper to learn more.
And if these frontiers of AI systems excite you, feel free to reach out to us!
1 Trainium2 was run on a synthetic multi-turn dataset (70k ISL, 3.5k new ISL 95% cached, 700 OSL) instead of AgentX since our implementation does not yet support higher concurrencies on the limited HBM capacity.
Together Compiler: Continual Compilation for AI inference
To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!
ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
Together AI at ICML 2026: frontier research across the full stack
ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)
Violin: An open-source video translation skill that breaks language barriers
Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding
Parcae: Doing more with fewer parameters using stable looped models
EinsteinArena: Harnessing the collective intelligence of agents in the wild to advance science
AI for Systems: Using LLMs to Optimize Database Query Execution
Inside the Together AI kernels team
Plan, divide, and conquer: How weak models excel at long context tasks
Key research and product announcements at the AI Native Conf
FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving
CoderForge-Preview: SOTA open dataset for training efficient coding agents
How speech models fail where it matters the most and what to do about it
Consistency diffusion language models: Up to 14x faster inference without sacrificing quality
What do LLMs think when you don't tell them what to think about?
DSGym: A holistic framework for evaluating and training data science agents
Research POV: Yes, AGI Can Happen – A Computational Perspective
How to run TorchForge reinforcement learning pipelines in the Together AI Native Cloud
Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation
Large Reasoning Models Fail to Follow Instructions During Reasoning: A Benchmark Study
AdapTive-LeArning Speculator System (ATLAS): A New Paradigm in LLM Inference via Runtime-Learning Accelerators
How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference Systems
Back to The Future: Evaluating AI Agents on Predicting Future Events
DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL
From Zero to One: Building An Autonomous and Open Data Scientist Agent from Scratch
Model-Preserving Adaptive Rounding with YAQA
Mixture-of-Agents Alignment: Harnessing the Collective Intelligence of Open-Source LLMs to Improve Post-Training
Boosting DeepSeek-R1’s Speed with Customized Speculative Decoding
Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas
Direct Preference Optimization: A Technical Deep Dive
Continued Fine-tuning of LLMs: A Technical Deep Dive
DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level
ThunderKittens Now Optimized for NVIDIA Blackwell GPUs
Minions: embracing small LMs, shifting compute on-device, and cutting cloud costs in the process
Long Context Fine-Tuning: A Technical Deep Dive
Fine-Tuning LLMs for Multi-Turn Conversations: A Technical Deep Dive
Even Better, Even Faster Quantized LLMs with QTIP
Linearizing LLMs with LoLCATs
Multimodal Document RAG with Llama 3.2 Vision and ColQwen2
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
Speculative decoding for high-throughput long-context inference
TEAL: Training-Free Activation Sparsity in Large Language Models
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
Building a personalized code assistant with open-source LLMs using RAG Fine-tuning
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
Together MoA — collective intelligence of open-source models pushing the frontier of LLM capabilities
Dragonfly: A large vision-language model with multi-resolution zoom
ThunderKittens: A Simple Embedded DSL for AI kernels
FAQ: Building LLMs with RedPajama-v2, a 30 trillion token web dataset
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
BASED: Simple linear attention language models balance the recall-throughput tradeoff
Evo: Long-context modeling from molecular to genome scale
BitDelta: Your Fine-Tune May Only Be Worth One Bit
Long context retrieval models with Monarch Mixer
Mamba-3B-SlimPJ: State-space models rivaling the best Transformer architecture
Paving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond Transformers
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
RedPajama-Data-v2: An open dataset with 30 trillion tokens for training large language models
Flash-Decoding for long-context inference
Medusa: Simple framework for accelerating LLM generation with multiple decoding heads
Llama-2-7B-32K-Instruct — and fine-tuning for Llama-2 models with Together API
Faster inference enables up to 5x price reduction on Together API
Preparing for the era of 32K context: Early learnings and explorations
Monarch Mixer: A new model architecture for increased efficiency
Fine-tuning language models over slow networks using activation compression with guarantees
Decentralized training of foundation models in heterogeneous environments
FlashAttention: Fast and memory-efficient exact attention with IO-Awareness
CocktailSGD: Fine-tuning foundation models over 500Mbps networks
FlexGen: High-throughput generative inference of large language models with a single GPU
Hyena Hierarchy: Towards larger convolutional language models
FlashConv: Speeding up state space models
Hungry Hungry Hippos: Towards language modeling with state space models
NeurIPS 2022: Overcoming communication bottlenecks for decentralized training (2/2)
NeurIPS 2022: Overcoming communication bottlenecks for decentralized training (1/2)
HELM: benchmarking large language models on the Together Research Computer
Together Compiler: Continual Compilation for AI inference
From Static to Polymorphic Inference: A new era for production open-source inference systems
Auto repair for Together GPU Clusters: from GPU fault to healthy node
Together Post-Training: frontier-grade model shaping, the way that fits you
Meta Muse Spark Open is live on Together AI
Expanding our enterprise inference capacity with IBM Cloud and NVIDIA
Together Link: frontier-quality open models in the harness you already use
How to train your own Jev for $17
Canary rollouts: upgrade models in production without downtime
How a global fintech scaled coding agent traffic with Dedicated Model Inference
Migrating from closed to open source models, Together
Together AI expands fine-tuning service with more models, live metrics, and finer controls
原文为英文,正文由自动抓取并保留原文。
原文发布于 Together AI Blog(Together Compiler: Continual Compilation for AI inference),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。