⚡ LLMsID: moe-routing

MoE Routing Top-k

Q: How does MoE routing work — what does top-k selection mean?

【MoE Routing Top-k 核心考点 1】How does MoE routing work — what does top-k selection mean?。技术解析：Definition: MoE (Mixture-of-Experts) places N experts in FFN layers; a gating network g(x) = \text{softmax}(W_g x) scores every token and selects the top-k experts, outputting the weighted sum y = \sum_{i \in \text{top-}k} g_i(x) E_i(x); DeepSeek-V3, for instance, has 671B total ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/moe-routing

Q: Why 671B total but 37B active — what do active params mean?

【MoE Routing Top-k 核心考点 2】Why 671B total but 37B active — what do active params mean?。技术解析：Definition: MoE (Mixture-of-Experts) places N experts in FFN layers; a gating network g(x) = \text{softmax}(W_g x) scores every token and selects the top-k experts, outputting the weighted sum y = \sum_{i \in \text{top-}k} g_i(x) E_i(x); DeepSeek-V3, for instance, has 671B total ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/moe-routing

Q: What is the gating network's input and how is it trained?

【MoE Routing Top-k 核心考点 3】What is the gating network's input and how is it trained?。技术解析：Definition: MoE (Mixture-of-Experts) places N experts in FFN layers; a gating network g(x) = \text{softmax}(W_g x) scores every token and selects the top-k experts, outputting the weighted sum y = \sum_{i \in \text{top-}k} g_i(x) E_i(x); DeepSeek-V3, for instance, has 671B total ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/moe-routing

Q: Trade-offs between top-1 and top-2 routing?

【MoE Routing Top-k 核心考点 4】Trade-offs between top-1 and top-2 routing?。技术解析：Definition: MoE (Mixture-of-Experts) places N experts in FFN layers; a gating network g(x) = \text{softmax}(W_g x) scores every token and selects the top-k experts, outputting the weighted sum y = \sum_{i \in \text{top-}k} g_i(x) E_i(x); DeepSeek-V3, for instance, has 671B total ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/moe-routing

Q: Why does MoE cut both training and inference cost?

【MoE Routing Top-k 核心考点 5】Why does MoE cut both training and inference cost?。技术解析：Definition: MoE (Mixture-of-Experts) places N experts in FFN layers; a gating network g(x) = \text{softmax}(W_g x) scores every token and selects the top-k experts, outputting the weighted sum y = \sum_{i \in \text{top-}k} g_i(x) E_i(x); DeepSeek-V3, for instance, has 671B total ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/moe-routing

MoE 路由 Top-k

🎯Core Definition

MoE (Mixture-of-Experts) places

N

experts in FFN layers; a gating network

g(x) = \text{softmax}(W_g x)

scores every token and selects the top-

k

experts, outputting the weighted sum

y = \sum_{i \in \text{top-}k} g_i(x) E_i(x)

; DeepSeek-V3, for instance, has 671B total parameters but activates only 37B per token — sparse activation is the core mechanism.

💡Use Cases

large-scale pretraining — buy more capacity with more parameters under the same FLOPs budget; inference only computes routed experts, cutting per-token compute; Llama 4, DeepSeek, and Mixtral all use top-2 or larger top-k routing.

⚡Key Problems Solved

scaling a Dense model scales compute proportionally; MoE breaks the params-≈-compute coupling via conditional computation, letting total parameters exceed active parameters by a large factor and boosting capacity at a fixed compute budget.

🎯Core Interview Exam Points

Fundamentals · Production Scenarios · Deep Dive

Core Def

Explain the core definition, underlying mechanism/math, and key principles of MoE Routing Top-k.

Use Cases

In production engineering, what are the primary use cases for MoE Routing Top-k and what core pain points does it solve?

🔥 High-Frequency Deep-Dive Questions

How does MoE routing work — what does top-k selection mean?

Why 671B total but 37B active — what do active params mean?

What is the gating network's input and how is it trained?

Trade-offs between top-1 and top-2 routing?

Why does MoE cut both training and inference cost?