Dense vs MoE

Dense models activate all parameters per token; MoE activates only a subset of expert parameters per token (sparse activation). At the same active-parameter count (e.g., 37B active), MoE's total parameter count can be several times a Dense model's (DeepSeek-V3: 671B total / 37B active) — more capacity at the same compute; but MoE must store all expert weights, incurs all-to-all communication between experts, and costs more memory and scheduling complexity at serving time, with harder downstream compression (quantization/distillation).

MoE for pretraining when the compute budget is fixed but capability ceiling matters; Dense for latency-sensitive, memory-constrained, or maintenance-cost-sensitive mid/small deployments; in practice MoE is often re-compressed via distillation/quantization or balanced with shared-expert hybrid designs.

Dense models hit a capacity ceiling under a fixed training-FLOP budget; MoE trades sparse activation for larger capacity but shifts the cost from training compute to serving memory and engineering complexity — the choice is fundamentally a capacity-vs-serving-cost trade-off.

Capability difference between MoE and Dense at the same active-param count, and why?

Why is serving MoE more expensive?

What is MoE's all-to-all communication cost?

When should you pick Dense over MoE?

How is MoE compressed via quantization/distillation?