11-19 更新 实时
首页/ 技术栈/ 正文

各 MoE 模型采用哪种路由?DeepSeek、Qwen、Mixtral、Llama 4、gpt-oss 对比

技术栈 来源:MarkTechPost 2026-11-08 18:27 4 阅读 机器翻译

开放混合专家(MoE)模型共享同一个理念,但在几乎每一个路由细节上都不一致。专家数量从 8 到 384 不等。Top-k 从 1 到 8 不等。有些模型保留一个常开的共享专家;另一些则将其去掉。

下面每个模型都采用 Token 选择路由:每个 Token 自行选择专家。它们在 5 个可调项上存在差异,分别是专家数量、每个 Token 的专家数、共享专家、打分函数和负载均衡方法。表格列出了每个模型的各项设置。

来源:各模型的 Hugging Face config.json、模型卡或技术报告,于 2026 年 11 月 7 日核实。Qwen3.6 和 Llama 4 的配置文件与模型卡未说明门控方式。gpt-oss 和 Llama 4 未说明均衡方法。

DeepSeek-V3 用 sigmoid 为 256 个路由专家打分,然后对选中的分数做归一化。每个专家的偏置用于引导 top-8 选择,但不影响混合权重。专家分为 8 组,路由器先保留 4 组。根据技术报告,每个 Token 最多到达 4 个节点。每个 Token 都会经过一个共享专家。

DeepSeek-V4 将 Pro 的路由专家数提升到 384,但把 top-k 从 8 降到 6。打分函数变为 sqrtsoftplus。前 3 个 MoE 层使用哈希路由,即固定的 Token 到专家映射。基于偏置的 noaux_tc 均衡方法保持不变。

最初的两个 Qwen3 MoE 模型都使用 128 个专家,并将每个 Token 路由到 8 个专家。Qwen3 取消了 Qwen2.5-MoE 的共享专家,改用全局批次均衡损失。Qwen3.6-35B-A3B 改变了方向:根据其模型卡,使用 256 个专家,8 个路由专家加 1 个共享专家。

Llama 4 Maverick 交替使用稠密层和 MoE 层。在每个 MoE 层中,一个 Token 会进入 1 个共享专家和 128 个路由专家中的 1 个。这种 top-1 设计呼应了 Switch Transformer。尽管总参数量达 400B,其激活参数量仍保持 17B。

gpt-oss-120b 在 128 个专家之上使用线性路由器,并保留 top 4。其配置中未列出共享专家。其 MoE 权重以 MXFP4 格式发布,这使得这个 117B 模型可以装进单张 80 GB GPU。

Kimi K2 采用 DeepseekV3ForCausalLM 架构类和 noaux_tc 路由器。它在同样 top-8 的情况下把路由专家数提升到 384。Kimi K2 的报告发现,在固定激活计算量下,更高的稀疏度可降低损失。GLM-4.5 则选择了 160 个专家和更多层,偏向深度而非宽度。

每层更多专家意味着更多权重常驻内存,以及更多的 all-to-all 通信流量。更低的 top-k(如 DeepSeek-V4 那样)可削减每个 Token 的计算与通信开销。共享专家增加了一条每个 Token 都要付出代价的稠密路径。在为任何 MoE 部署规划 GPU 规模之前,先检查这些字段。

模型来源:DeepSeek-V4-Pro、DeepSeek-V3 报告、Qwen3 报告、Kimi K2 报告、GLM-4.5 报告、Mixtral 论文。另见 gpt-oss-120b 和 Llama 4 公告。

需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等?欢迎联系我们

文章《每个 MoE 模型分别使用哪种路由?DeepSeek、Qwen、Mixtral、Llama 4、gpt-oss 对比》首发于 MarkTechPost。

译文由英文原文机器翻译,术语按行业惯例统一,可在文末展开原文核对。

查看英文原文(Which Routing Does Each MoE Model Use? DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss Compared)

Open Mixture-of-Experts (MoE) models share one idea but disagree on almost every routing detail. Expert counts range from 8 to 384. Top-k ranges from 1 to 8. Some models keep an always-on shared expert; others drop it.

TL;DR

  • Size: 10 models, from Qwen3.6-35B-A3B (35B total, 3B active) to DeepSeek-V4-Pro (1.6T total, 49B active).
  • Runs on: gpt-oss-120b fits one 80 GB GPU with MXFP4; DeepSeek-V4-Pro and Kimi K2 need multi-GPU expert parallelism.
  • Performance: Routed sparsity spans 4x in Mixtral (8 experts, top-2) to 128x in Llama 4 Maverick (128, top-1).
  • Best: Kimi K2 found sparsity 48 needed 1.69x fewer FLOPs than sparsity 8 at equal validation loss.
  • Worst: Llama 4 Maverick and gpt-oss do not publish their gating or balancing method in config.json.
  • Bottom line: Best thing: configs make routing fully auditable. Worst thing: names like “17B active” hide very different routers.

How do open MoE models route tokens?

Every model below uses token-choice routing: each token selects its own experts. They differ on 5 knobs. These are expert count, experts per token, shared experts, the scoring function and the load-balancing method. The table lists each knob per model.

MoE routing comparison table (November 2026)

ModelTotal / active paramsRouted expertsTop-kShared expertsGatingLoad balancing
Mixtral 8x7B47B / 13B820Softmax over top-kAux loss, coef 0.02 (config)
gpt-oss-120b117B / 5.1B12840Softmax over selectedNot disclosed
Qwen3-235B-A22B235B / 22B12880Softmax, top-k normalizedGlobal-batch aux loss, coef 0.001
Qwen3.6-35B-A3B35B / 3B25681Not disclosedAux loss, coef 0.001 (config)
Llama 4 Maverick400B / 17B12811Not disclosedNot disclosed
DeepSeek-V3671B / 37B25681SigmoidAux-loss-free bias (noaux_tc)
DeepSeek-V4-Pro1.6T / 49B38461sqrtsoftplusnoaux_tc; hash routing in 3 layers
DeepSeek-V4-Flash284B / 13B25661sqrtsoftplusnoaux_tc; hash routing in 3 layers
Kimi K21.04T / 32B38481Sigmoidnoaux_tc + sequence aux, alpha 0.001
GLM-4.5355B / 32B16081SigmoidLoss-free balance routing

Sources: each model’s Hugging Face config.json, model card or technical report, verified November 7, 2026. Configs and cards for Qwen3.6 and Llama 4 omit gating. gpt-oss and Llama 4 omit the balancing method.

How does DeepSeek-V3 routing work?

DeepSeek-V3 scores 256 routed experts with a sigmoid, then normalizes the selected scores. A per-expert bias steers the top-8 selection without touching the mixing weights. Experts sit in 8 groups, and the router keeps 4 groups first. Each token reaches at most 4 nodes, according to the technical report. One shared expert runs for every token.

What changed in DeepSeek-V4 routing?

DeepSeek-V4 raises Pro to 384 routed experts but lowers top-k from 8 to 6. The score function becomes sqrtsoftplus. The first 3 MoE layers use hash routing, a fixed token-to-expert map. Bias-based noaux_tc balancing stays.

How many experts does Qwen3 MoE use?

Both original Qwen3 MoE models use 128 experts and route each token to 8. Qwen3 dropped the shared experts of Qwen2.5-MoE and adopted a global-batch balancing loss. Qwen3.6-35B-A3B changes course: 256 experts, 8 routed plus 1 shared, per its model card.

How does Llama 4 Maverick routing differ?

Llama 4 Maverick alternates dense and MoE layers. In each MoE layer, a token goes to 1 shared expert and 1 of 128 routed experts. That top-1 design echoes the Switch Transformer. It keeps 17B active parameters despite 400B in total.

How does gpt-oss route tokens?

gpt-oss-120b uses a linear router over 128 experts and keeps the top 4. Its config lists no shared expert. Its MoE weights ship in MXFP4, which lets the 117B model fit on a single 80 GB GPU.

Why do Kimi K2 and GLM-4.5 look like DeepSeek-V3?

Kimi K2 ships with the DeepseekV3ForCausalLM architecture class and the noaux_tc router. It raises routed experts to 384 at the same top-8. The Kimi K2 report found that higher sparsity lowered loss at fixed active compute. GLM-4.5 chose 160 experts and more layers instead, favoring depth over width.

What the trend means for serving

More experts per layer means more weights resident in memory and more all-to-all traffic. Lower top-k, as in DeepSeek-V4, trims per-token compute and communication. Shared experts add a dense path that every token pays for. Check these fields before sizing GPUs for any MoE deployment.

Key Takeaways

  • Expert counts grew from 8 (Mixtral) to 384 (Kimi K2, DeepSeek-V4-Pro).
  • Top-k now spans 1 (Llama 4) to 8 (DeepSeek-V3, Qwen3, Kimi K2, GLM-4.5).
  • 7 of 10 models keep 1 shared expert; Mixtral, gpt-oss and Qwen3 do not.
  • Sigmoid or sqrtsoftplus with bias balancing is the DeepSeek-family standard.
  • Llama 4 and gpt-oss leave gating or balancing details unpublished.


Model sources: DeepSeek-V4-Pro, DeepSeek-V3 report, Qwen3 report, Kimi K2 report, GLM-4.5 report, Mixtral paper. Also see gpt-oss-120b and the Llama 4 announcement.

Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post Which Routing Does Each MoE Model Use? DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss Compared appeared first on MarkTechPost.

正文保留英文原文,标题与摘要由机器翻译。

本文由采集器自动同步

原文发布于 MarkTechPost(Which Routing Does Each MoE Model Use? DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss Compared),标题、摘要与正文由机器翻译,术语按行业惯例统一,未经人工逐句校订。

阅读原文 ↗