M3-079M3: Deep Learning FoundationsDistributed Training BasicsEasy
⚡Flashcard Drill➔Mastery:

ZeRO (Zero Redundancy Optimizer): The Three Sharding Stages Explained

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-080 →
📐 Mathematical Definition
ZeRO-1: shard optimizer;ZeRO-2: +grad;ZeRO-3: +param\text{ZeRO-1}:\ \text{shard optimizer};\quad \text{ZeRO-2}:\ +\text{grad};\quad \text{ZeRO-3}:\ +\text{param}
⚡ One-Sentence Core Answer
ZeRO eliminates memory redundancies in distributed training by sharding optimizer states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across data-parallel ranks.

📌 Key Takeaways

  • •
    ZeRO-1: shards 16-byte FP32 optimizer states (Adam mm and vv); reduces memory by 4×4\times with zero extra communication
  • •
    ZeRO-2: shards gradients; ranks reduce-scatter gradients and retain only their partition; reduces memory by 2×2\times
  • •
    ZeRO-3 (FSDP): shards model parameters; gathers full weights on-the-fly via All-Gather before each layer forward/backward, and discards them immediately

📐 Mathematical Derivations

Memory Breakdown and Sharding Mechanics (Rajbhandari et al., DeepSpeed, SC 2020):
For a model with Ψ\Psi parameters trained in FP16 mixed-precision with Adam:
- Model Weights (FP16): 2Ψ2 \Psi bytes.
- Gradients (FP16): 2Ψ2 \Psi bytes.
- Optimizer States (FP32 master weights 4Ψ4 \Psi, momentum mm 4Ψ4 \Psi, variance vv 4Ψ4 \Psi): 12Ψ12 \Psi bytes.
Total baseline memory per GPU: 2Ψ+2Ψ+12Ψ=16Ψ2\Psi + 2\Psi + 12\Psi = 16\Psi bytes (e.g., 256GB for a 16B parameter model).
ZeRO Stages across $N$ Data-Parallel Ranks:
① ZeRO-1 ($P_{\text{os}}$): Shards optimizer states across NN ranks. Memory becomes 2Ψ+2Ψ+12ΨN2\Psi + 2\Psi + \frac{12\Psi}{N}. Communication: exact same All-Reduce overhead as standard DDP.
② ZeRO-2 ($P_{\text{os+g}}$): Shards both optimizer states and gradients. Gradients are synchronized via Reduce-Scatter instead of All-Reduce. Each rank updates only its 1/N1/N parameter partition. Memory: 2Ψ+2ΨN+12ΨN2\Psi + \frac{2\Psi}{N} + \frac{12\Psi}{N}. Communication: identical volume to standard DDP (2Ψ2\Psi).
③ ZeRO-3 ($P_{\text{os+g+p}}$): Shards model parameters as well. Each rank stores only Ψ/N\Psi / N parameters. Memory drops to 16ΨN\frac{16\Psi}{N} bytes. Prior to each layer's forward pass, ranks perform an All-Gather to reconstruct parameters, compute layer, and release weights. Backward pass executes a second All-Gather. Increases communication volume by 1.5×1.5\times.

🏭 Production Trade-offs

System design trade-offs: ZeRO-1/2 provide substantial memory relief with zero additional network traffic. ZeRO-3 allows training multi-billion parameter models without Tensor Parallelism, but incurs 50% more communication, requiring high-bandwidth interconnects (InfiniBand/RoCE).
⚠️ Common Interview Pitfalls
  • ✕
    Using ZeRO-3 on clusters with slow 10Gbps Ethernet, causing training to be 90% communication-bound
  • ✕
    Assuming ZeRO-1 increases network communication volume; ZeRO-1 has mathematically identical communication to PyTorch DDP
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does ZeRO-3 require 50% more communication volume than standard Data Parallelism (DDP)?
  • ?
    How does ZeRO-Offload leverage host CPU RAM and NVMe SSDs to train massive models on single GPUs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-078: Comparing Distributed Parallelism: Data (DP), Tensor (TP), and Pipeline (PP)📋 Back to BankNext →M3-080: Collective Communication Primitives: All-Reduce, All-Gather, and Reduce-Scatter