11-19 更新 实时
首页/ 技术栈/ 正文

MoE 路由策略:Top-K、Sigmoid 与动态路由

技术栈 来源:MarkTechPost 2026-11-08 14:18 0 阅读 机器翻译

每个混合专家(MoE)模型都要为每个 Token 重复做出同一个决策:该由哪些专家来处理它。这个决策由路由器负责。在每一个主流开放模型中,MoE 路由策略都影响着质量、负载均衡和服务成本。本指南将解释生产环境中使用的 7 种路由策略,以及各自对应的模型。

MoE 路由是一种机制,把每个 Token 发送给一小部分专家网络。路由器会为每个专家打分,保留得分最高的候选者,并对其输出加权。只有被选中的专家才会运行。例如 Mixtral 8x7B 让每个 Token 可以访问 47B 参数,但实际只使用 13B 激活参数。

每个 Token 通过一个线性层为所有专家打分,并保留得分最高的 k 个。对这 k 个分数做 softmax,即得到混合权重。Mixtral 从 8 个专家中选 2 个。gpt-oss-120b 从 128 个中选 4 个。Qwen3-235B-A22B 从 128 个中选 8 个。通常还会用一项辅助损失来保持专家负载均衡。

采用者:Mixtral、gpt-oss、Qwen3。

Switch Transformer 表明,把每个 Token 只路由到 1 个专家,既能保持质量,又能削减路由器的计算量。每个专家会获得一个固定缓冲区:每批 Token 数除以专家数,再乘以容量因子。溢出的 Token 通过残差连接跳过该层。Llama 4 Maverick 同样把每个 Token 发送给 1 个被路由的专家,另加一个共享专家。

采用者:Switch Transformer、Llama 4 Maverick(搭配一个共享专家)。

DeepSeekMoE 将每一层拆分为始终在线的共享专家和稀疏路由的专家。共享专家负责捕捉通用知识,从而让被路由的专家可以专注特化。DeepSeek-V3 使用 1 个共享专家和 256 个路由专家,激活其中 8 个。Kimi K2、GLM-4.5 和 Qwen3.6-35B-A3B 沿用同样的模式。Qwen3 则反其道而行,取消了共享专家。

采用者:DeepSeek-V3 和 V4、Kimi K2、GLM-4.5、Qwen3.6、Llama 4。

DeepSeek-V3 用 sigmoid 而非 softmax 为专家打分。它只在选择 top-k 时加入每个专家的偏置。每一步之后,负载过重的专家偏置下降,空闲专家的偏置上升。混合权重仍来自原始得分。GLM-4.5 使用带 sigmoid 门控的无损失均衡路由。DeepSeek-V4 把打分改为 sqrtsoftplus,但保留了偏置均衡。

采用者:DeepSeek-V3、DeepSeek-V4、GLM-4.5、Kimi K2。

把一个 Token 发送到多个节点上的专家代价高昂。DeepSeek-V3 将每个 Token 限制在 4 个节点内,按亲和度得分最高的节点来选。其配置把专家分成 8 组,在最终挑出 top-8 之前先保留 4 组。DeepSeek 表示,这一限制让训练中的计算与通信几乎可以完全重叠。Kimi K2 则完全取消分组,将 n_group 设为 1。

采用者:DeepSeek-V2 和 V3。

Google 的专家选择路由把方向颠倒过来。每个专家挑选自己的 top-k Token,因此每个专家得到固定的桶大小。Token 可以接收数量不定的专家。论文报告训练收敛速度提升超过 2 倍。问题在于因果性:一个 Token 的专家可能依赖于后面的 Token。DeepSeek 的研究人员测到这种泄漏带来的异常损失下降约为 10%。

采用者:研究和非因果场景。我们对照表中的 10 个模型均未采用。

哈希路由根据 Token ID 分配专家,不需要学习型路由器。DeepSeek-V4 在其前 3 个 MoE 层使用哈希路由。动态 top-k 让每个 Token 激活的专家数量可变。SMIDT 把它集成进 SGLang,将 P99 首 Token 延迟改善了 1.21 到 1.26 倍。

采用者:DeepSeek-V4(哈希路由)、研究型服务系统(动态 top-k)。

大多数团队并不会主动挑选路由器。他们是在选定 checkpoint 时顺带继承了其中一个。在部署服务之前,先读一读 config.json 里的 4 个字段:专家数量(expert count)、每个 Token 激活的专家数(experts per token)、共享专家(shared experts)以及 topk_method。在推理阶段改动 top-k 等于改动了模型本身,所以要把它当作一次质量实验来对待。对于新的预训练任务,带偏置平衡的 sigmoid 打分是目前公开的默认做法。

需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等?欢迎联系我们

文章《MoE 路由策略详解:Top-K、Expert-Choice、Sigmoid 门控与动态路由(2026 指南)》最初发布于 MarkTechPost。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide))

Every Mixture-of-Experts (MoE) model repeats one decision for every token: which experts should process it. That decision belongs to the router. MoE routing strategies shape quality, load balance and serving cost in every major open model. This guide explains the 7 routing strategies used in production, and the models that use each one.

TL;DR

  • Size: Routers choose from 8 experts (Mixtral) to 384 (DeepSeek-V4-Pro), activating 1 to 8 per token.
  • Runs on: Routing cost shows up in multi-GPU expert parallelism, where each choice becomes all-to-all network traffic.
  • Performance: Sigmoid scoring with aux-loss-free bias balancing is the dominant 2026 recipe for frontier open MoE models.
  • Best: Loss-free balancing cut MaxVio from 0.72 to 0.04 at 1B, with lower perplexity.
  • Worst: Expert choice leaks future tokens in causal models; one test showed a roughly 10% abnormal loss drop.
  • Bottom line: Best thing: bias-based balancing adds no interfering gradients. Worst thing: no router removes the need for serving-time load fixes.

What is MoE routing?

MoE routing is the mechanism that sends each token to a small subset of expert networks. A router scores every expert, keeps the top candidates and weights their outputs. Only the selected experts run. Mixtral 8x7B, for example, gives each token access to 47B parameters but uses 13B active parameters.

video created by marktechpost.com

The 7 MoE routing strategies used in production

1. Softmax top-k token choice

Each token scores all experts with a linear layer and keeps the k highest. A softmax over those k scores sets the mixing weights. Mixtral picks 2 of 8 experts. gpt-oss-120b picks 4 of 128. Qwen3-235B-A22B picks 8 of 128. An auxiliary loss usually keeps expert load even.

Used by: Mixtral, gpt-oss, Qwen3.

2. Top-1 switch routing with a capacity factor

Switch Transformer showed that routing each token to 1 expert preserves quality and cuts router compute. Each expert gets a fixed buffer: tokens per batch divided by experts, times a capacity factor. Overflow tokens skip the layer through the residual connection. Llama 4 Maverick also sends each token to 1 routed expert, plus a shared expert.

Used by: Switch Transformer, Llama 4 Maverick (with a shared expert).

3. Shared plus routed experts

DeepSeekMoE splits each layer into always-on shared experts and sparsely routed experts. The shared expert captures common knowledge, so routed experts can specialize. DeepSeek-V3 uses 1 shared and 256 routed experts, activating 8. Kimi K2, GLM-4.5 and Qwen3.6-35B-A3B follow the same pattern. Qwen3 went the other way and removed shared experts.

Used by: DeepSeek-V3 and V4, Kimi K2, GLM-4.5, Qwen3.6, Llama 4.

4. Sigmoid gating with aux-loss-free bias

DeepSeek-V3 scores experts with a sigmoid instead of a softmax. It adds a per-expert bias only when choosing the top-k. After each step, the bias drops for overloaded experts and rises for idle ones. Mixing weights still come from the raw scores. GLM-4.5 uses loss-free balance routing with sigmoid gates. DeepSeek-V4 switches the score to sqrtsoftplus but keeps bias balancing.

Used by: DeepSeek-V3, DeepSeek-V4, GLM-4.5, Kimi K2.

5. Group-limited and node-limited routing

Sending one token to experts on many nodes is expensive. DeepSeek-V3 caps each token at 4 nodes, chosen by their highest affinity scores. Its config splits experts into 8 groups and keeps 4 before the final top-8 pick. DeepSeek says this cap lets training nearly fully overlap computation and communication. Kimi K2 drops grouping entirely, with n_group set to 1.

Used by: DeepSeek-V2 and V3.

6. Expert choice routing

Google’s expert choice routing flips the direction. Each expert picks its top-k tokens, so every expert gets a fixed bucket size. Tokens can receive a variable number of experts. The paper reports more than 2x faster training convergence. The catch is causality: a token’s expert can depend on later tokens. DeepSeek researchers measured an abnormal loss drop of about 10% from this leakage.

Used by: Research and non-causal settings. None of the 10 models in our companion table uses it.

7. Hash routing and dynamic top-k

Hash routing assigns experts from the token ID, with no learned router. DeepSeek-V4 uses hash routing in its first 3 MoE layers. Dynamic top-k lets the number of active experts vary per token. SMIDT built it into SGLang and improved P99 time-to-first-token by 1.21 to 1.26x.

Used by: DeepSeek-V4 (hash routing), research serving systems (dynamic top-k).

MoE routing strategies compared

StrategyWho choosesHow load is balancedMain riskUsed by
Softmax top-kToken picks k expertsAuxiliary lossLoss trades off qualityMixtral, gpt-oss, Qwen3
Top-1 switchToken picks 1 expertAux loss + capacity factorDropped overflow tokensSwitch, Llama 4 Maverick
Shared + routedToken picks routed; shared always runsInherits routed methodExtra always-on computeDeepSeek, Kimi K2, GLM-4.5, Qwen3.6
Sigmoid + biasToken picks via biased scorePer-expert bias, no loss gradientImbalance inside one sequenceDeepSeek-V3/V4, GLM-4.5, Kimi K2
Node-limitedToken picks within capped nodesInherits routed methodRestricts expert choiceDeepSeek-V2/V3
Expert choiceExpert picks k tokensFixed by constructionFuture-token leakageResearch models
Hash / dynamicToken ID or variable kFixed mapping or adaptive kLess learned specializationDeepSeek-V4, SMIDT

Which MoE routing strategy should you use?

Most teams do not pick a router. They inherit one when they choose a checkpoint. Before serving, read 4 fields in config.json: expert count, experts per token, shared experts and topk_method. Changing top-k at inference changes the model itself, so treat it as a quality experiment. For new pretraining runs, sigmoid scoring with bias balancing is the current open default.

Key Takeaways

  • Top-k token choice is the baseline; k ranges from 1 (Llama 4) to 8 (DeepSeek-V3, Qwen3).
  • Shared experts are standard in DeepSeek, Kimi, GLM and Qwen3.6; Qwen3 dropped them.
  • Sigmoid plus bias balancing is the 2026 default for frontier open MoE models.
  • Expert choice balances perfectly but leaks future tokens in causal LLMs.
  • DeepSeek-V4 adds hash routing to its first 3 MoE layers.


Check out the primary sources: Switch Transformer, Expert Choice, Loss-Free Balancing, DeepSeek-V3 report and the DeepSeek-V4 model card. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide) appeared first on MarkTechPost.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 MarkTechPost(MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide)),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗