11-19 更新 实时
首页/ 技术栈/ 正文

Polymorphic Inference:开放模型自调优推理

技术栈 来源:Together AI Blog 2026-11-04 08:00 0 阅读 机器翻译
Polymorphic Inference:开放模型自调优推理

推出 Together Post-Training:更好的模型、更快的运行、更低的成本 →

推出 Together Link:在你已使用的 harness 中运行开放模型 →

开源 AI 技术栈:模型、推理、路由器、harness 与工具 →

GLM-5.3 与 GLM 5.3 Flash 在 DeepSWE 上的对比 →

⚡ 按需 B200 现已在 Together GPU 集群上线 →

以 API 形式提供的高性能推理

面向批处理工作负载的推理

基于 Token 的容量,附带 SLA

专用模型推理

定制硬件上的推理

专用容器推理

自定义模型推理

在你自己的测试框架中运行开放模型

随时间不断优化的推理

探索顶级开源模型

大规模可靠 GPU 集群

前沿规模的自定义基础设施

为 AI 构建开发环境

安全存储模型权重与数据

定义你自己的强化学习(RL)与监督微调(SFT)循环

启动托管的 SFT 与 DPO 任务

微调顶尖开源模型

面向生产级 AI 的系统研究

我们全部的研究出版物

Together AI 技术文档

我们的最新新闻与博客文章

我们的开源演示应用

实用实施指南

用开放模型构建更好的成果

在你自己的 harness 中运行开放模型

浏览我们的活动日历

开源 ROI 计算器

以 API 形式提供的高性能推理

面向批处理工作负载的推理

基于 Token 的容量与 SLA

专用模型推理

在自定义硬件上推理

专用容器推理

面向自定义模型的推理

在自有运行框架中运行开放模型

随时间不断改进的推理

探索顶级开源模型

大规模可靠的 GPU 集群

前沿规模的自定义基础设施

为 AI 构建开发环境

安全存储模型权重与数据

定义你自己的 RL 和 SFT 循环

启动托管的 SFT 和 DPO 任务

微调顶级开源模型

面向生产级 AI 的系统研究

我们的全部研究出版物

Together AI 技术文档

我们的最新新闻与博客文章

我们的开源演示应用

实用实施指南

用开放模型构建更好的应用

在你的运行框架中运行开放模型

探索我们的活动日历

开源 ROI 计算器

从静态推理到多态推理:生产级开源推理系统的新时代

从今天起,你通过 Together AI 端点运行的许多模型,都已由 AI 智能体完成全面优化:这是全球首个生产规模的递归自我改进(RSI)推理系统。

在 Together AI,我们致力于让 AI 惠及每一个用户和组织。我们的研究团队——FlashAttention、ThunderKittens、ThunderAgent 等背后的研究者——一直是我们推理卓越表现的关键推动力。随着推理市场呈指数级增长,我们一直在思考如何扩展自身的研究能力,以实现普惠的智能。

这要求我们的思路从静态推理转向多态推理(Polymorphic Inference)——这是我们为应对新模型发布的速度、模型架构的多样性、快速变化的工作负载以及快速变化的硬件平台而给出的答案。在本篇博客中,我们将介绍多态推理时代的研究议程,以及我们如何持续利用 AI 智能体,通过持续编译(Continuous Compilation)来优化自身的推理。

什么是多态推理?

过去一年,开源模型的多样性以及这些模型在生产环境中的部署方式都出现了显著的阶跃式变化。我们认为,这种阶跃式变化需要一种新的方法,我们称之为「多态推理」(Polymorphic Inference)。

多态推理(Polymorphic Inference,源自希腊语,意为「多种形态」的推理)是对我们在整个行业中所看到的一系列变化的回应,对我们而言,这些变化主要由几股趋势的汇聚所定义:

模型发布速度不断加快。开源模型正以比以往任何时候都更快的节奏推出——而且多样性远超以往。OpenRouter 上 80% 的流量流向了过去三个月内发布的模型。

模型架构的多样性不断提升。目前使用中最受欢迎的十大 OSS 模型全都采用不同的架构(更不用说我们合作伙伴所服务的众多闭源架构了)。

流量形态持续演变。我们每天为新的用户、组织和地区提供服务,覆盖广泛的工作负载形态和 SLA。编排框架(harness)的产品或算法变更、开源模型的后期训练(post-training)都可能导致流量形态发生变化(例如,一次新的后期训练增加了模型用于思考的 Token 数量,以换取更高的质量)。

硬件更新更快、更频繁。从 B200 到 GB300,我们看到推理服务能力发生了重大变化,并且即将迎来首批 Vera Rubin 部署(更不用说 CPU 层面的优化、先进存储系统和替代性推理芯片了)。

多态推理是我们对这些变化的回应——这种推理会随这些快速变化的环境而改变并适应。

我们将这一点与 OSS 推理爆发之前的世界状态进行对比——我们回溯性地将其称为“静态推理”时代。现代多态推理呈现出无穷无尽的不同形态,而静态推理则由闭源模型、传统 Transformer 架构、单用户以及单一硬件平台推理来定义。

从静态推理到多态推理

我们撰写这篇博客文章的动机之一是,围绕推理系统的许多讨论仍然相当聚焦于遗留的静态推理范式。

例如,网上典型的 vLLM、SGLang 或 InferenceX / AgentX 方案,都会以静态超参数调优过的推理引擎配置来启动。针对一种场景优化过的配置,未必是另一种场景的最佳配置。

开源内核在发布时常常假定某种静态流量形态和设计决策,而这些可能与真实生产流量失配。一个典型例子是 MoE 内核;在随机 Token 上表现良好的 MoE 内核,未必能迁移到真实生产场景。

我们在这里也要承担一部分责任!长期以来,写出能在多种场景下最优的内核一直是可能的。FlashAttention 是我们十分珍视的成果,它仍然是运行训练负载和许多预填充负载的最佳方式之一。

而大约两年前,我们曾撰文介绍用于 AI 推理的巨型内核(megakernel),即围绕单一静态形状(例如 batch size 为 1 的 Llama 1B)精心设计的单个内核。但去年我们为部分客户将首批巨型内核投入生产的过程中,得到了一个有趣的教训:对于现代多态推理(Polymorphic Inference)而言,巨型内核融合其实并非正确的设计目标!在多态推理时代,每一种变化——从模型架构(例如 MoE)到工作负载(大于 batch size 1 的动态负载)——都需要不同的技术路径。

为多样化、快速变化的条件优化多态推理

在 Together,我们骨子里是系统与 AI 研究者——因此我们认为,在这个崭新未来重新思考我们的研究议程,是一个极其激动人心的时刻。

那么,当你的生产流量变得日益多样化时,解决方案是什么?让 AI 智能体自我优化。这便是开源(OSS)模型的 RSI 时刻!

实现多态推理需要在模型服务方式上解决一系列技术挑战。理想的多态推理体验,应当能够应对模型架构与工作负载快速变化的格局。

如果你的流量形态发生变化——无论是产品变更还是用户行为改变——你都应期待推理系统能快速适应。

如果你要在新模型上部署工作负载——你应期待推理服务商尽快逼近理想的手工调优基线。

如果你已有用户流量稳定的现有产品,并将其部署到新的硬件平台——你应期待性能尽快逼近硬件极限。

这些目标很宏大,但我们相信它们值得为之努力。对我们而言,实现这些目标的关键之一,是让 AI 智能体帮助优化它们自身。递归自我改进(RSI)循环已经来到开源模型身边!

基于 Together Compiler 的持续编译

我们今天公布的众多项目中,有一个是 Together Compiler——一个由 AI 驱动、可快速进行性能优化的 RSI 循环,它是我们迈向 Polymorphic Inference 目标的第一步。如今,Together Compiler 已在我们许多顶级生产推理负载上驱动持续编译优化循环。

Together Compiler 把优化推理负载这一经典难题,从传统的编译器翻译问题重新定义为智能体式搜索问题。Together Compiler 识别出一组核心的关键优化模式,并将一群智能体与一个高保真、持久化的引擎配对,以实现快速迭代。这项工作由 Together 的 Frontier Performance 团队历时数月完成,主要贡献者是 Reyna Abhyankar 和 Dylan Lim!

原文较长,译文为节选;译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Polymorphic Inference: self-tuning inference for open models)

Introducing Together Post-Training: Better models, faster runs, lower costs →

Introducing Together Link: Run open models inside the harness you already use →

The Open Source AI Stack: models, inference, routers, harnesses & tools →

GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →

⚡ On-demand B200s now available on Together GPU Clusters →

High-performance inference as APIs

Inference for batch workloads

Token-based capacity with SLAs

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

Run open models in your harness

Inference that gets better over time

Explore Top Open-Source Models

Reliable GPU clusters at scale

Custom infrastructure at frontier scale

Build development environments for AI

Store model weights & data securely

Define your own RL and SFT loop

Launch managed SFT and DPO jobs

Fine-tune top open-source models

Systems research for production AI

All our research publications

Technical docs for Together AI

Our latest news & blog posts

Our open-source demo apps

Practical implementation guides

Build better with open models

Run open models in your harness

Explore our events calendar

Open-source ROI calculator

High-performance inference as APIs

Inference for batch workloads

Token-based capacity with SLAs

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

Run open models in your harness

Inference that gets better over time

Explore Top Open-Source Models

Reliable GPU clusters at scale

Custom infrastructure at frontier scale

Build development environments for AI

Store model weights & data securely

Define your own RL and SFT loop

Launch managed SFT and DPO jobs

Fine-tune top open-source models

Systems research for production AI

All our research publications

Technical docs for Together AI

Our latest news & blog posts

Our open-source demo apps

Practical implementation guides

Build better with open models

Run open models in your harness

Explore our events calendar

Open-source ROI calculator

From Static to Polymorphic Inference: A new era for production open-source inference systems

As of today, many of the models you run through Together AI endpoints are fully optimized by AI agents: the world’s first production-scale recursive self-improvement (RSI) inference system.

At Together AI, we focus on making AI abundantly available to every individual user and organization. Our research team – the researchers behind FlashAttention, ThunderKittens, ThunderAgent, and more – has been a key driver behind our inference excellence. As the inference market grows exponentially, we’ve been focusing on how to scale our research capacity to enable abundant intelligence for all.

This has required a shift in our thinking from Static Inference to Polymorphic Inference – which is our response to the velocity of new model releases, diversity of model architectures, fast-changing workloads, and fast-changing hardware platforms. In this blog, we’ll highlight our research agenda for the Polymorphic Inference era – and how we are continuously using AI agents to optimize their own inference with Continuous Compilation.

What is Polymorphic Inference?

Over the past year, there has been a marked step change in the diversity of open-source models and the ways that these models are deployed in production environments. We see this step change as requiring a new approach that we call Polymorphic Inference.

Polymorphic Inference (inference in “many shapes” from Greek) is a response to a number of changes we are seeing across the industry, for us primarily defined by the convergence of a few trends:

Increasing velocity of model releases. Open-source models are coming out at a faster clip – and are far more diverse – than ever before. 80% of traffic on OpenRouter goes to models released in the last three months.

Increasing diversity of model architectures. All of the top-ten most popular OSS models in use today use different architectures (not to mention the many closed-source architectures served by our partners).

Traffic shapes that continuously evolve. We serve new users, organizations, and geographies every day across a broad range of workload shapes and SLAs. Product or algorithmic changes in harnesses, post-training of open-source models can all lead to changes to traffic shapes (e.g., a new post-train that increases the amount of tokens that a model spends thinking in exchange for higher quality).

Hardware that is changing faster and more frequently. We have seen major changes in inference serving capabilities from B200 to GB300, and are on the cusp of seeing the first Vera Rubin deployments (not to mention CPU-level optimizations, advanced storage systems, and alternative inference chips).

Polymorphic Inference is our response to these changes – inference that changes and adapts to these rapidly-changing environments.

We contrast this against the state of the world before the explosion of OSS inference – which we retroactively call the era of “Static Inference.” Whereas modern Polymorphic Inference takes on infinite different shapes, Static Inference is defined by closed models, traditional Transformer architectures, single-user, and single-hardware-platform inference.

From Static to Polymorphic Inference

One of our motivations behind writing this blog post is that a lot of the discourse around inference systems remains quite focused on the legacy Static Inference paradigm.

For example, the typical vLLM, SGLang, or InferenceX / AgentX recipe online is launched with a static hyperparameter-tuned inference engine configuration. A config optimized for one setting is not necessarily the best one for another setting.

Open-source kernels are frequently released assuming some static traffic shapes and design decisions that can be miscalibrated against real production traffic. A classic example is MoE kernels; an MoE kernel that works well against random tokens may not transfer to real production settings.

We take some of the blame here! For a long time, it has been possible to write kernels that are optimal for a variety of settings. FlashAttention, near and dear to our hearts, is still one of the best ways to run training workloads and many prefill workloads.

And nearly two years ago we wrote about megakernels for AI inference, a single kernel painstakingly designed around a single static shape (e.g., batch size one Llama 1B). But through the experience of putting some of the first megakernels into production for our customers last year, we learned an interesting lesson: megakernel fusion is actually not the right design goal for modern Polymorphic Inference! In the era of Polymorphic Inference, every variation – from model architecture (e.g. MoE), to workload (dynamic workloads larger than batch size 1) – requires different technical approaches.

Optimizing Polymorphic Inference for diverse, rapidly-changing conditions

At Together, we are systems and AI researchers at heart – so we view this as an incredibly exciting time to be rethinking our research agenda for this bright new future.

‍So, what’s the solution when your production traffic becomes ever more diverse? Get the AI agents to optimize themselves. This is the RSI moment for OSS models!

Achieving Polymorphic Inference requires solving a number of technical challenges in how we serve models. An ideal Polymorphic Inference experience is one that responds to the rapidly changing landscape of model architectures and workloads.

If your traffic shape changes – either through a product change or a change in user behavior – you should expect your inference system to adapt quickly.

If you are deploying workloads on a new model – you should expect your inference provider to approach ideal hand-tuned baselines as soon as possible.

If you have an existing product with stable user traffic, and you deploy it on a new hardware platform – you should expect performance to approach hardware bounds as quickly as possible.

These are lofty goals, but we believe that they are worth striving for. For us, one of the keys to achieving these goals is getting AI agents to help optimize themselves. The RSI loop is here for OSS models!

Continuous Compilation with Together Compiler

One of the projects that we’re announcing today as a first step towards achieving our goals for Polymorphic Inference is Together Compiler, a rapid AI-powered performance optimization RSI loop. Today, Together Compiler powers Continuous Compilation optimization loops across many of our top production inference workloads.

Together Compiler takes the classic challenge of optimizing inference workloads and recasts it from a traditional compiler translation problem to an agentic search problem. Together Compiler identifies a core set of key optimization patterns and pairs a swarm of agents with a high-fidelity persistent engine for fast iterations. This work was the result of many months of work by Together’s Frontier Performance team, notably from Reyna Abhyankar and Dylan Lim!

Together Compiler is running in production today on some of our most popular serverless endpoints, where it has found further gains on top of already highly optimized code.

Excitingly, Together Compiler gives us the ability to quickly adapt to rapidly changing model architectures and traffic workloads. We call this capability Continuous Compilation. Whereas previously new workloads would require new sets of hand-written kernels and optimizations, Continuous Compilation driven by Together Compiler means that we can automatically adapt to new traffic shapes and workloads.

What’s coming next from Together AI’s Research team

We’ve learned a ton scaling Together AI to serve trillions of tokens per day (and for the first time even serving more tokens than closed-source model labs on OpenRouter earlier in October!). This has informed our vision and research agenda for the Polymorphic Inference era. Here’s a little preview of some of the things coming up on our research agenda:

Self-serve Continuous Compilation with Together Compiler‍

Performance optimization needs to be a function of the workload deployment: the model, engine, traffic shape, and customer use case. Soon, our customers will be able to use Together Compiler to optimize their dedicated endpoints for their unique workloads at the click of a button.

Together Compiler is available to end customers in limited beta today – reach out if you want early access!

Shape-shifting speculative decoding

Inference engine configurations proposed in open-source repositories, VLLM, InferenceX, AgentX launch the engine with a fixed speculative decoding draft model and lookahead parameter. Speculative decoding needs to adapt per-domain, per-request, per-token and as a function of the traffic load.

ThunderAgent Router: Routing across heterogeneous deployments

Serverless inference endpoints serve an enormous number of users, each bringing their own traffic shapes and SLAs. This heterogeneity requires us to dynamically route requests across workers and deployment configurations across our clusters. We released ThunderAgent as an ICML 2026 Spotlight paper to share some early insights and have expanded on these foundations in production.

Several customers now want to incorporate open-source model backends into their AI applications. This requires choosing an open-source model, selecting and customizing the harness to the customer’s specific tool calls and use cases, and post-training the model to interact well with this harness. Our close partnerships with model labs including Moonshot, Minimax, and others allows us to help our customers customize open-source models for their own use cases and really own their intelligence stack.

TogetherLink: Workload-aware model routing

As OSS models improve, a key piece is getting them into everyone’s hands. The common perception is that a closed source model like Opus 5.5 is always best. But, as the history of AI has repeatedly shown us, ensembling a few pretty good models can definitely outperform using a singular great model, if combined in the right way. This builds on our roots in weak supervision at Stanford’s Hazy Research Lab (Ask Me Anything) and on our own work on combining models (Mixture-of-Agents, ICLR 2025).

Abundant intelligence for all

Inference is one of the most fascinating problems in AI Systems and we’ve unlocked a whole host of new insights through the scale at which we serve. To achieve our goal of making AI abundant for everyone we need our inference systems to continuously customize to each user and each workload in the Polymorphic Inference era. We’re excited to help close the loop and play a small role in making AI systems truly recursively self-improving.

If the ideas behind Polymorphic Inference ring true to your organization or your use case, please reach out for early access  – we’d love to help you navigate these new complexities.

And lastly – if you want to help us tackle the challenge of Polymorphic Inference together, please take a look at our careers page – we’re hiring!

From Static to Polymorphic Inference: A new era for production open-source inference systems

Expanding our enterprise inference capacity with IBM Cloud and NVIDIA

Together AI announces strategic partnership with Moonshot AI to natively serve Kimi models

Announcing our $800M Series C to accelerate the shift to open-source AI

Building trust in enterprise AI: Together AI earns ISO 27001:2022 certification

From 732 bytes to nowhere: shutting down Copy Fail in production

What is an AI Native Cloud?

Together AI at NVIDIA GTC 2026: Explore our latest innovations across research and products

Introducing Together AI’s new look

Together AI welcomes Alon Gavrielov as VP of Infrastructure Strategy

Announcing Together Python SDK v2.0

Together AI and Meta partner to bring PyTorch Reinforcement Learning to the AI Native Cloud

Announcing the Together AI Startup Accelerator, purpose-built for AI Native Apps

Together AI welcomes Mahadev Konar as SVP for Infrastructure Engineering

Powering Secure AI: Together AI Achieves SOC 2 Type 2 Compliance

Together AI acquires Refuel.ai to unlock data for developers and businesses building production-grade AI applications

From AWS to Together Dedicated Endpoints: Arcee AI's journey to greater inference flexibility

Scaling AI Companions: How Dippy AI Reached Over 4 Million Tokens/Minute with Together Dedicated Endpoints

Together AI Awarded ClusterMAX™ Gold Rating by SemiAnalysis

Together AI becomes NVIDIA Cloud Partner to bolster accelerated AI offerings

Together AI Announces $305M Series B to Scale AI Acceleration Cloud for Open Source and Enterprise AI

Together AI acquires CodeSandbox to launch first-of-its-kind code interpreter for generative AI

AWS Marketplace now offering Together AI to accelerate enterprise AI development

Together AI to Co-Build Turbocharged NVIDIA GB200 Cluster with 36K Blackwell GPUs in Partnership with Hypertec Cloud

How Zomato built an AI customer support bot that doubled customer satisfaction and scaled to over 1,000 messages per minute

Introducing The Together Enterprise Platform: Run GenAI securely in any environment, with 2x faster inference and continuous model optimization

Together AI welcomes Kai Mak as CRO to accelerate gen AI adoption for AI natives and enterprises globally

Together AI and NVIDIA collaborate to power Llama 3.1 models for enterprises on NVIDIA DGX Cloud

Flash Attention received the inaugural Stanford open source software award

Together AI partners with Snowflake to bring Arctic LLM to Enterprise customers

Announcing $106M round led by Salesforce Ventures

Introducing Together AI Chief Scientist Tri Dao, as he releases FlashAttention-2 to speed up model training and inference

Together AI and Snorkel AI empower enterprises to build proprietary LLMs

Together’s $20M seed funding to build open-source AI and cloud platform

How Together and Crusoe are reducing the carbon impact of generative AI

Together Compiler: Continual Compilation for AI inference

From Static to Polymorphic Inference: A new era for production open-source inference systems

Auto repair for Together GPU Clusters: from GPU fault to healthy node

Together Post-Training: frontier-grade model shaping, the way that fits you

Meta Muse Spark Open is live on Together AI

Expanding our enterprise inference capacity with IBM Cloud and NVIDIA

Together Link: frontier-quality open models in the harness you already use

How to train your own Jev for $17

Canary rollouts: upgrade models in production without downtime

How a global fintech scaled coding agent traffic with Dedicated Model Inference

Migrating from closed to open source models, Together

Together AI expands fine-tuning service with more models, live metrics, and finer controls

原文为英文,正文由自动抓取并保留原文。

本文由采集器自动同步

原文发布于 Together AI Blog(Polymorphic Inference: self-tuning inference for open models),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗