M4-074M4: Sequences & TransformersMixture of Experts (MoE)Easy
⚡Flashcard Drill➔Mastery:
Mixture of Experts (MoE) Architecture and Sparse Activation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Mixture of Experts replaces dense feed-forward networks with multiple parallel expert sub-networks, using a gating router to conditionally activate only the top- experts per token, vastly scaling parameter capacity without increasing compute FLOPs.
📌 Key Takeaways
- •Sparse activation: separates total parameter capacity (e.g., 8x7B = 47B parameters) from active parameters per token (e.g., 2 experts = 13B FLOPs)
- •Structure: retains standard self-attention; replaces the standard FFN in each Transformer block with independent FFN experts and a gating router
- •Advantages: achieves the representation capacity of massive models while maintaining the inference and training latency of much smaller dense models
📐 Mathematical Derivations
Mathematical Mechanism: 1. MoE Layer Formulation: Given input token representation and expert networks (where each is a standard FFN): The gating router computes logits (with optional Gaussian noise during training). 2. Top-$k$ Routing: To enforce sparse execution, only the top- experts are selected (, typically or ): Where if is in the top values of , and otherwise. 3. Output Computation: The final output is the gate-weighted combination of the selected active experts: Because only out of experts evaluate , the computational complexity is rather than , achieving constant FLOPs scaling with respect to total expert count .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① FLOPs vs VRAM Footprint: While compute FLOPs match a small dense model (e.g., Mixtral 8x7B uses FLOPs), all experts must reside in GPU memory (demanding VRAM in FP16), requiring multi-GPU expert parallelism (EP). ② Routing Collapse: Without regularization, the router exhibits rich-get-richer dynamics: a few favored experts receive all tokens, while other experts receive zero gradient updates and die out (solved by auxiliary load balancing loss). ③ Expert Parallelism All-to-All Overhead: When experts are distributed across GPUs, routing tokens requires global
all-to-all communication to dispatch tokens to their target GPUs and gather the results, creating severe network communication bottlenecks. ④ Serving Memory Bandwidth: At low batch sizes during decoding, every active request may select different experts, forcing the GPU to load weights from all experts and reducing effective memory bandwidth efficiency. ⑤ Interview Strategy: Write down the top- gating equation, explain the distinction between total parameters and active FLOPs, and highlight the memory and communication trade-offs.⚠️ Common Interview Pitfalls
- ✕Confusing total parameter count with active compute FLOPs per token
- ✕Assuming MoE reduces memory footprint (it vastly increases VRAM requirements compared to an equivalent-compute dense model)
- ✕Overlooking router collapse where only a tiny fraction of experts are actually utilized
🎯 Interviewer Follow-up Inquiries
- ?Why does low batch size inference degrade MoE throughput compared to dense models?
- ?How does expert capacity factor prevent memory overflow when too many tokens route to the same expert?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚