M4-078M4: Sequences & TransformersMixture of Experts (MoE)Hard
⚡Flashcard Drill➔Mastery:
Fine-Tuning and Deployment Challenges of MoE Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
MoE fine-tuning and deployment are complicated by routing instability on small task-specific datasets, high distributed memory requirements, and fragmented kernel execution during serving.
📌 Key Takeaways
- •Fine-tuning instability: task-specific data skews token distribution, leading to severe router imbalance, expert starvation, or catastrophic forgetting of specialized domains
- •PEFT / LoRA choices: applying LoRA to all experts expands parameter count significantly; applying LoRA only to attention or shared routers can limit adaptation capacity
- •Deployment infrastructure: requires multi-GPU expert parallelism, complex kernel scheduling (grouped GEMMs / Cutlass), and careful load balancing across serving replicas
📐 Mathematical Derivations
Mathematical Mechanism: 1. Router Distribution Shift under Fine-Tuning: Pre-trained router weights are calibrated to general pre-training token distributions. During downstream fine-tuning on a narrow domain (e.g., medical QA or Python code), token features cluster in a small subspace: If expert happens to score highest on , receives of all domain tokens. The auxiliary load balancing loss fights against this specialization: - If is high, it forces code tokens to route artificially to math or poetry experts. - If is low, overfits while other experts receive zero updates and undergo catastrophic forgetting. 2. LoRA Adaptation Formulations: For each expert : With experts, LoRA parameters scale as , which can exceed the parameter count of a standard dense LoRA by an order of magnitude.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Freezing the Router vs Updating Router: Freezing router weights during fine-tuning preserves pre-trained routing stability and prevents collapse, but prevents the model from discovering new expert specializations for novel tasks. Common practice is to freeze for small datasets and train with small learning rates on large instruction datasets. ② LoRA Placement Strategy: Applying LoRA exclusively to attention weights () avoids multi-expert LoRA complexity while delivering 85-90% of full fine-tuning performance. Alternatively, applying LoRA to top- experts or shared experts strikes an efficient middle ground. ③ Inference Engine Kernels: Grouped GEMM: Standard GEMM kernels assume uniform matrix dimensions. In MoE, each expert receives a dynamically varying number of tokens (). Executing separate GEMM kernels per expert incurs severe GPU launch latency. Grouped GEMM (CUTLASS / Triton) executes all variable-batch GEMMs in a single fused kernel launch. ④ Quantization Difficulties: MoE weights exhibit distinct activation outlier channels across different experts, making uniform post-training quantization (PTQ) prone to severe accuracy drops unless per-expert quantization scales are used. ⑤ Interview Strategy: Detail the router distribution shift problem during domain fine-tuning, compare LoRA placement options, and explain Grouped GEMM serving acceleration.
⚠️ Common Interview Pitfalls
- ✕Fine-tuning MoE on a small domain dataset with standard training hyperparameters, leading to immediate routing collapse
- ✕Launching separate GEMM kernels sequentially for each expert during serving instead of using fused Grouped GEMM
- ✕Assuming LoRA on MoE is always lightweight without accounting for the multiplier across all experts
🎯 Interviewer Follow-up Inquiries
- ?Why is fused Grouped GEMM critical for accelerating MoE inference on GPUs?
- ?What happens to router allocation when an MoE model is fine-tuned with frozen router weights versus trainable router weights?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚