11-19 更新 实时
首页/ 技术栈/ 正文

Together Compiler:面向 AI 推理的持续编译

技术栈 来源:Together AI Blog 2026-11-04 08:00 0 阅读 机器翻译
Together Compiler:面向 AI 推理的持续编译

推出 Together Post-Training:模型更优、运行更快、成本更低 →

推出 Together Link:在您已在使用的运行框架内运行开放模型 →

开源 AI 技术栈:模型、推理、路由器、运行框架与工具 →

GLM-5.3 与 GLM 5.3 Flash 在 DeepSWE 上的对比 →

⚡ 按需 B200 现已在 Together GPU 集群上线 →

以 API 形式提供的高性能推理

面向批处理工作负载的推理

带 SLA 的按 Token 计费容量

专用模型推理

自定义硬件上的推理

专用容器推理

自定义模型的推理

在自有运行框架中运行开放模型

随时间不断优化的推理

探索顶尖开源模型

大规模可靠 GPU 集群

前沿规模下的定制基础设施

为 AI 构建开发环境

安全存储模型权重与数据

定义你自己的 RL 与 SFT 循环

启动托管式 SFT 和 DPO 任务

微调顶级开源模型

面向生产级 AI 的系统研究

我们的全部研究出版物

Together AI 技术文档

我们的最新新闻与博客文章

我们的开源演示应用

实用实施指南

用开放模型构建更优方案

在你的测试框架中运行开放模型

浏览我们的活动日历

开源 ROI 计算器

以 API 形式提供的高性能推理

面向批处理工作负载的推理

带 SLA 的基于 Token 的容量

专属模型推理

在自定义硬件上运行推理

专属容器推理

面向自定义模型的推理

在你自己的运行框架中运行开放模型

随时间持续提升的推理

探索顶级开源模型

规模化部署的可靠 GPU 集群

前沿规模的自定义基础设施

为 AI 构建开发环境

安全存储模型权重与数据

定义你自己的 RL 与 SFT 循环

启动托管式 SFT 与 DPO 任务

微调顶级开源模型

面向生产环境 AI 的系统研究

我们的全部研究论文

Together AI 技术文档

我们的最新新闻与博客文章

我们的开源演示应用

实用实施指南

用开放模型构建更好的应用

在你的 harness 中运行开放模型

浏览我们的活动日历

开源 ROI 计算器

Together Compiler:面向 AI 推理的持续编译

AI 推理工作负载是动态的。存在数千种工况,涵盖新模型、硬件平台、客户、用例和流量形态。再靠人工逐一优化这些工况,已不再具备可扩展性。

架构高度多样:距 Transformer 问世已过去约 11 年。我们或许以为会出现更多整合收敛,但 OpenRouter 上排名前 10 的开源 LLM 无一采用相同架构!

工作负载高度多样:Together 每天为数千家客户服务数万亿 Token,每一家的流量形态都不同(批大小、序列长度、MoE 专家不均衡、注意力稀疏度、会话时长、领域)。

硬件高度多样:H100、B200、B300、NVL72、VR、Groq 等各自都需要不同的策略。

理论上,智能体非常适合解决这个问题。它们能够适应新的工作负载,并在可能的优化空间中进行搜索,而不必依赖那些困扰人类专家和传统机器学习编译器的固定规则或启发式方法。然而,如今的智能体式系统受制于实验速度,形成了瓶颈。

我们推出 Together Compiler(TKCC),这是首个真正持续运行的优化循环,利用高吞吐智能体面向多样化的推理工作负载进行优化。

在不到 24 小时内,TKCC 在前沿开源模型(Kimi K3、GLM 5.3、MiniMax M3.1-Flash、DeepSeek v4.1-Flash)上实现了最高 2.6 倍的 TPS 提升,在 SemiAnalysis 的 InferenceX 低延迟基准测试中刷新了最优成绩——MiniMax M3.1-Flash 达到 1111 TPS,DeepSeek v4.1-Flash 达到 1205 TPS,并在其主力场景 Llama 1B、batch size 1 下,比现有的开源 megakernel 快 1.7 倍。此外,TKCC 还能泛化到语言模型推理之外的领域(天气建模、PDE、时间序列和生物学)。这一性能提升得益于 TKCC 相比基线智能体式性能优化系统高出 252 倍的实验吞吐。如今,Together Compiler 已在服务数万亿 Token。

流量在不断变化

作为推理服务提供商,我们看到用户行为多种多样,这些行为受到地域模式、新模型发布和突发性的影响。

地域模式:世界各地的用户可能有不同的模型偏好。我们观察到,某国工作时段流量的提示词长度是另一国的 1.4 倍(97k vs 70k)。

模型发布:用户需要时间才能弄清如何把新发布的模型整合进自己的工作流。起初,他们往往选择基于对话的交互与评估。而发布仅两周后,随着用户转向智能体式工作负载,提示词长度会迅速增加 80%。

突发性:客户可能在一天中的任意时段发送突发请求。提示词长度可能突然翻倍,如果处理不当,缓存命中率也会下降。

显然,我们必须针对每一次新的部署做专门的优化,以适应用户行为的不断变化。

Together Compiler 概览

TKCC 以流量形态、模型和硬件作为输入。TKCC 使用智能体为推理引擎生成一系列优化。

图 1:随着流量漂移,Together Compiler 会重新编译,智能体、常驻引擎与集成体构成一个循环。

TKCC 的核心洞察如下:

1. TKCC 在智能体优化过程中最大化实验吞吐,并最小化 GPU 空闲时间

大多数现有的智能体式推理优化系统要求智能体启动和拆除 VLLM、SGLang 或 TRT-LLM 等引擎,以测试每个新实验。然而,引擎启动时间长达数十分钟,严重阻碍了实验吞吐(见下表)。相反,TKCC 提出了一种“持久化引擎”的新范式:引擎只需启动一次,就能热重载任何候选前向传播,无需任何重启。为了确保持久化引擎始终有新的优化可测试,TKCC 会创建多个智能体,持续生成想法,并通过留言板通信,以并行推进各自的工作。

RPC 调用 + Together Compiler 设置

表 1:引擎启动成本高昂

2. TKCC 平衡测量吞吐与测量保真度之间的权衡

现有的智能体优化系统要么反复重新启动推理引擎,因而实验吞吐低下;要么在推理引擎之外孤立地优化独立内核,因而测量保真度差。优化独立内核的系统通常使用随机输入数据来评估它们。然而,现代架构使用 MoE 和稀疏 k 注意力等依赖数据的原语,输入数据分布会直接影响哪些优化策略可能胜出(见下图)。高保真输入对于有效优化很重要。TKCC 使用分层持久化引擎来平衡吞吐与保真度之间的权衡:“fast engines”加载少量层,适合快速迭代和低方差计时;而“slower engines”加载许多层,适合测试优化能否在真实数据分布上成立。

图 2:输入数据会改变优化策略。

图 3:Together Compiler 的两层架构优于其他方案。

3. TKCC 利用自定义成本模型为智能体选择并行策略(如 TP、EP、DCP 等),以实现最高性能。

现有系统已有成熟的方法,可在满足 SLA 要求的前提下选择推理配置。然而,选择理想配置面临同样的吞吐与保真度权衡:逐一测量每种并行配置是不现实的,尤其是在软件实现尚不成熟的情况下。一种朴素的做法是依据硬件的原始规格或微基准测试来选择并行策略。但这种方式无法充分刻画依赖数据的流量特征,例如 MoE 专家负载不均衡或 prompt 长度分布不均。我们设计了一个成本模型,采用分层测量策略——结合硬件规格、微基准测试和实测前向传播——来构建高保真的表征,同时快速剪枝配置搜索空间,以支持持续的流量漂移。

图 4:实测布局优于硬件规格所选布局,且最佳布局随并发变化而改变。

我们将 TKCC 与三类基线进行对比评估:手工调优的 MegaKernel、InferenceX 的引擎配置,以及来自气象建模、PDE、时间序列和生物模型等不同领域的流行 AI 代码库。

我们首先评估经典的 MegaKernel 工作负载:批大小为 1 的 Llama 1B。该模型架构已被人专家和传统机器学习编译器充分优化。TKCC 实现了 1.7 倍的 TPS 提升,并且随着序列长度增加仍能保持性能优势。

图 5:Together Compiler 在 H100 和 B200 上均优于 MegaKernel 和 CUDA graph 基线。

InferenceX AgentX 工作负载

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Together Compiler: Continual Compilation for AI inference)

Introducing Together Post-Training: Better models, faster runs, lower costs →

Introducing Together Link: Run open models inside the harness you already use →

The Open Source AI Stack: models, inference, routers, harnesses & tools →

GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →

⚡ On-demand B200s now available on Together GPU Clusters →

High-performance inference as APIs

Inference for batch workloads

Token-based capacity with SLAs

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

Run open models in your harness

Inference that gets better over time

Explore Top Open-Source Models

Reliable GPU clusters at scale

Custom infrastructure at frontier scale

Build development environments for AI

Store model weights & data securely

Define your own RL and SFT loop

Launch managed SFT and DPO jobs

Fine-tune top open-source models

Systems research for production AI

All our research publications

Technical docs for Together AI

Our latest news & blog posts

Our open-source demo apps

Practical implementation guides

Build better with open models

Run open models in your harness

Explore our events calendar

Open-source ROI calculator

High-performance inference as APIs

Inference for batch workloads

Token-based capacity with SLAs

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

Run open models in your harness

Inference that gets better over time

Explore Top Open-Source Models

Reliable GPU clusters at scale

Custom infrastructure at frontier scale

Build development environments for AI

Store model weights & data securely

Define your own RL and SFT loop

Launch managed SFT and DPO jobs

Fine-tune top open-source models

Systems research for production AI

All our research publications

Technical docs for Together AI

Our latest news & blog posts

Our open-source demo apps

Practical implementation guides

Build better with open models

Run open models in your harness

Explore our events calendar

Open-source ROI calculator

Together Compiler: Continual Compilation for AI inference

AI inference workloads are dynamic. There are thousands of regimes spanning new models, hardware platforms, customers, use cases, and traffic shapes. It’s no longer scalable to optimize each of these regimes by hand.

Architectures are highly variable: it’s been ~11 years since the Transformer. We might have thought there would be more consolidation, but none of the top-10 open-source LLMs on OpenRouter use the same architecture!

Workloads are highly variable: Together serves many trillions of tokens per day across thousands of customers, each with a different traffic shape (batch size, sequence length, MoE expert imbalance, attention sparsity, session duration, domain).

Hardware is highly variable: H100s, B200s, B300s, NVL72, VR, Groq, and more each require distinct strategies.

Agents are a great fit for this problem, in theory. They can adapt to new workloads and search over the space of possible optimizations without relying on fixed rules or heuristics that plague human experts and traditional machine learning compilers. However, today’s agentic systems are bottlenecked by the speed of experimentation.

We introduce Together Compiler (TKCC), the first trulycontinual optimization loop that leverages high-throughput agents to optimize to diverse inference workloads

In less than 24 hours, TKCC achieves up to 2.6x higher TPS across frontier open source models (Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash), sets the state-of-the-art on SemiAnalysis’ InferenceX benchmark for low-latency with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash, and outperforms available open-source megakernels by 1.7x on their workhorse Llama 1B batch size 1 setting. Furthermore, TKCC generalizes to domains beyond language model inference (weather modeling, PDE, time-series, and biology). This performance boost is enabled by TKCC’s 252x higher experiment throughput compared to baseline agentic performance optimization systems. Together Compiler is serving trillions of tokens today.

Traffic is continually changing

As an inference provider, we see a variety of user behaviors based on regional patterns, new model releases, and burstiness.

Regional patterns: Users across the world may have different model preferences. We’ve observed traffic for one country’s working hours to have 1.4x longer prompts than another country (97k vs 70k).

Model releases: Users take time to figure out how to integrate new model releases into their workflows. First, they tend to opt for chat-based interaction and evaluation. This is swiftly followed by an 80% increase in prompt length just two weeks after the release as users shift towards agentic workloads.

Burstiness: Customers may send bursts of requests at random times throughout the day. Prompt lengths can suddenly double and cache hit rates can fall if not handled carefully.

It’s clear that we must adapt to ever-changing user behavior with specialized optimizations for each new deployment.

Overview of Together Compiler

TKCC takes as input the traffic shape, model, and hardware. TKCC uses agents to generate a series of optimizations for an inference engine.

Figure 1: Together Compiler recompiles as traffic drifts, with agents, a persistent engine and an ensemble in a loop.

The core insights in TKCC are as follows:

1. TKCC maximizes experimentation throughput and minimizes GPU idle time during agent optimization

‍Most existing agentic inference optimization systems require agents to launch and teardown engines like VLLM, SGLang, or TRT-LLM to test each new experiment. However, engine start times are tens of minutes, significantly impeding experimentation throughput (see table below). Instead, TKCC proposes a new paradigm of “persistent engines” where the engine starts up once and can hot-reload any candidate forward pass, without any restarts. To ensure that the persistent engines constantly have new optimizations to test, TKCC spawns multiple agents that continually generate ideas and communicate via a message board to parallelize their efforts.

RPC call + Together Compiler setup

Table 1: Engine startup is expensive

2. TKCC balances the tradeoff between measurement throughput and measurement fidelity

‍Existing agent optimization systems either iteratively relaunch the inference engine, suffering from low experiment throughput, or optimize standalone kernels in isolation, outside of the inference engine, suffering from poor measurement fidelity. Systems that optimize standalone kernels often evaluate them using random input data. However, with modern architectures using data-dependent primitives like MoE and sparse-k attention, input data distributions have a direct effect on which optimization strategies may win (see figure below). High-fidelity inputs are important for effective optimization. TKCC uses a tier of persistent engines to balance the throughput-fidelity tradeoff: “fast engines” load a few layers and are ideal for rapid iteration and low-variance timings, while “slower engines” load many layers and are ideal for testing whether optimizations hold-up on real data distributions.

Figure 2: Input data changes optimization strategies.

Figure 3: Together Compiler’s two-tiered architecture outperforms alternatives.

3. TKCC utilizes a custom cost model to select parallelism strategies (e.g., TP, EP, DCP, etc.) for agents to achieve the highest performance

Existing systems have sophisticated methods for selecting inference configurations under SLA requirements. However, selecting the ideal configuration faces the same throughput-fidelity tradeoff: it is intractable to measure each parallelism configuration, especially because software implementations are immature. A naive approach is to select a parallelism strategy based on raw hardware specs or micro-benchmarks. However, this insufficiently captures data dependent traffic properties like MoE expert imbalance or uneven prompt length distributions. We designed a cost model that uses a tiered measurement strategy involving hardware specs, microbenchmarks, and measured forward passes to construct a high-fidelity representation while rapidly pruning the configuration search space to support continual traffic drift.

Figure 4: Measured layouts beat the hardware spec’s pick, and the best layout shifts with concurrency.

We evaluate TKCC against three different types of baselines: Hand-tuned megakernels, InferenceX engine configurations, and popular AI codebases from diverse domains like weather modeling, PDE, time series and biology models.

We start by evaluating the canonical MegaKernel workload: Llama 1B at batch size 1. This model architecture has been well-optimized by both human experts and traditional machine learning compilers. TKCC generates 1.7x higher TPS and continues to perform as sequence length increases.

Figure 5: Together Compiler outperforms megakernel and CUDA graph baselines on both H100 and B200

InferenceX AgentX workloads

We evaluate TKCC on the AgentX benchmark from SemiAnalysis. We start from the best-reported open source engine configurations for frontier models across inference engines and hardware platforms. TKCC consistently sets the performance frontier across all settings, outperforming highly-optimized baselines by up to 2.6x higher TPS.1

We also evaluate specifically on the low-latency use case at batch size 1 and set the state-of-the-art performance with 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash. For these results, we use our cost model to select the parallelism configuration instead of using the InferenceX configuration. We observe speedups with this approach.

Figure 6: Together Compiler serves more tokens per GPU at the same speed per user on most models.

Figure 7: Together Compiler is faster per user than the best InferenceX configuration on all six model and chip pairs

We select popular open-source code bases across weather modeling, PDE, time-series, and biology. We show the geometric mean speedups of TKCC over simply using Claude Code with the /goal command, illustrating that the principles of TKCC generalize across domains. We have been using TKCC as a general purpose AI compiler beyond LLM inference.

Figure 8: Together Compiler finds larger speedups than Claude Code on three of four models

Polymorphic AI systems is the next frontier

For Together AI, the most exciting systems challenge has been Polymorphic Inference. Inference is no longer about optimizing for a single traffic shape for a single hardware platform and model, traffic is constantly and dynamically evolving, much like financial markets, and that dynamism has opened up a whole host of new research questions.

Check out our paper to learn more.

And if these frontiers of AI systems excite you, feel free to reach out to us!

1 Trainium2 was run on a synthetic multi-turn dataset (70k ISL, 3.5k new ISL 95% cached, 700 OSL) instead of AgentX since our implementation does not yet support higher concurrencies on the limited HBM capacity.

Together Compiler: Continual Compilation for AI inference

To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!

ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale

Together AI at ICML 2026: frontier research across the full stack

ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

Violin: An open-source video translation skill that breaks language barriers

Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding

Parcae: Doing more with fewer parameters using stable looped models

EinsteinArena: Harnessing the collective intelligence of agents in the wild to advance science

AI for Systems: Using LLMs to Optimize Database Query Execution

Inside the Together AI kernels team

Plan, divide, and conquer: How weak models excel at long context tasks

Key research and product announcements at the AI Native Conf

FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

Cache-aware prefill–decode disaggregation (CPD) for up to 40% faster long-context LLM serving

CoderForge-Preview: SOTA open dataset for training efficient coding agents

How speech models fail where it matters the most and what to do about it

Consistency diffusion language models: Up to 14x faster inference without sacrificing quality

What do LLMs think when you don't tell them what to think about?

DSGym: A holistic framework for evaluating and training data science agents

Research POV: Yes, AGI Can Happen – A Computational Perspective

How to run TorchForge reinforcement learning pipelines in the Together AI Native Cloud

Introducing AutoJudge: Streamlined inference acceleration via automated dataset curation

Large Reasoning Models Fail to Follow Instructions During Reasoning: A Benchmark Study

AdapTive-LeArning Speculator System (ATLAS): A New Paradigm in LLM Inference via Runtime-Learning Accelerators

How Together AI Uses AI Agents to Automate Complex Engineering Tasks: Lessons from Developing Efficient LLM Inference Systems

Back to The Future: Evaluating AI Agents on Predicting Future Events

DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL

From Zero to One: Building An Autonomous and Open Data Scientist Agent from Scratch

Model-Preserving Adaptive Rounding with YAQA

Mixture-of-Agents Alignment: Harnessing the Collective Intelligence of Open-Source LLMs to Improve Post-Training

Boosting DeepSeek-R1’s Speed with Customized Speculative Decoding

Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas

Direct Preference Optimization: A Technical Deep Dive

Continued Fine-tuning of LLMs: A Technical Deep Dive

DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level

ThunderKittens Now Optimized for NVIDIA Blackwell GPUs

Minions: embracing small LMs, shifting compute on-device, and cutting cloud costs in the process

Long Context Fine-Tuning: A Technical Deep Dive

Fine-Tuning LLMs for Multi-Turn Conversations: A Technical Deep Dive

Even Better, Even Faster Quantized LLMs with QTIP

Linearizing LLMs with LoLCATs

Multimodal Document RAG with Llama 3.2 Vision and ColQwen2

The Mamba in the Llama: Distilling and Accelerating Hybrid Models

Speculative decoding for high-throughput long-context inference

TEAL: Training-Free Activation Sparsity in Large Language Models

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Building a personalized code assistant with open-source LLMs using RAG Fine-tuning

SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

Together MoA — collective intelligence of open-source models pushing the frontier of LLM capabilities

Dragonfly: A large vision-language model with multi-resolution zoom

ThunderKittens: A Simple Embedded DSL for AI kernels

FAQ: Building LLMs with RedPajama-v2, a 30 trillion token web dataset

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

BASED: Simple linear attention language models balance the recall-throughput tradeoff

Evo: Long-context modeling from molecular to genome scale

BitDelta: Your Fine-Tune May Only Be Worth One Bit

Long context retrieval models with Monarch Mixer

Mamba-3B-SlimPJ: State-space models rivaling the best Transformer architecture

Paving the way to efficient architectures: StripedHyena-7B, open source models offering a glimpse into a world beyond Transformers

FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores

RedPajama-Data-v2: An open dataset with 30 trillion tokens for training large language models

Flash-Decoding for long-context inference

Medusa: Simple framework for accelerating LLM generation with multiple decoding heads

Llama-2-7B-32K-Instruct — and fine-tuning for Llama-2 models with Together API

Faster inference enables up to 5x price reduction on Together API

Preparing for the era of 32K context: Early learnings and explorations

Monarch Mixer: A new model architecture for increased efficiency

Fine-tuning language models over slow networks using activation compression with guarantees

Decentralized training of foundation models in heterogeneous environments

FlashAttention: Fast and memory-efficient exact attention with IO-Awareness

CocktailSGD: Fine-tuning foundation models over 500Mbps networks

FlexGen: High-throughput generative inference of large language models with a single GPU

Hyena Hierarchy: Towards larger convolutional language models

FlashConv: Speeding up state space models

Hungry Hungry Hippos: Towards language modeling with state space models

NeurIPS 2022: Overcoming communication bottlenecks for decentralized training (2/2)

NeurIPS 2022: Overcoming communication bottlenecks for decentralized training (1/2)

HELM: benchmarking large language models on the Together Research Computer

Together Compiler: Continual Compilation for AI inference

From Static to Polymorphic Inference: A new era for production open-source inference systems

Auto repair for Together GPU Clusters: from GPU fault to healthy node

Together Post-Training: frontier-grade model shaping, the way that fits you

Meta Muse Spark Open is live on Together AI

Expanding our enterprise inference capacity with IBM Cloud and NVIDIA

Together Link: frontier-quality open models in the harness you already use

How to train your own Jev for $17

Canary rollouts: upgrade models in production without downtime

How a global fintech scaled coding agent traffic with Dedicated Model Inference

Migrating from closed to open source models, Together

Together AI expands fine-tuning service with more models, live metrics, and finer controls

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Together AI Blog(Together Compiler: Continual Compilation for AI inference),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗