⚡ LLMsID:
moe-routingMoE Routing Top-k
MoE 路由 Top-k
🎯Core Definition
MoE (Mixture-of-Experts) places experts in FFN layers; a gating network scores every token and selects the top- experts, outputting the weighted sum ; DeepSeek-V3, for instance, has 671B total parameters but activates only 37B per token — sparse activation is the core mechanism.
💡Use Cases
large-scale pretraining — buy more capacity with more parameters under the same FLOPs budget; inference only computes routed experts, cutting per-token compute; Llama 4, DeepSeek, and Mixtral all use top-2 or larger top-k routing.
⚡Key Problems Solved
scaling a Dense model scales compute proportionally; MoE breaks the params-≈-compute coupling via conditional computation, letting total parameters exceed active parameters by a large factor and boosting capacity at a fixed compute budget.
🎯5 High-Frequency Exam Points
1
How does MoE routing work — what does top-k selection mean?
2
Why 671B total but 37B active — what do active params mean?
3
What is the gating network's input and how is it trained?
4
Trade-offs between top-1 and top-2 routing?
5
Why does MoE cut both training and inference cost?
📖 In-depth Guide:📄 moe-architecture →
Updated 2026-08-12🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "MoE Routing Top-k"
Single choice pitfall questions with instant feedback and mistake tracking.