M3-079M3: Deep Learning FoundationsDistributed Training BasicsEasy
⚡Flashcard Drill➔Mastery:
ZeRO (Zero Redundancy Optimizer): The Three Sharding Stages Explained
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
ZeRO eliminates memory redundancies in distributed training by sharding optimizer states (ZeRO-1), gradients (ZeRO-2), and model parameters (ZeRO-3) across data-parallel ranks.
📌 Key Takeaways
- •ZeRO-1: shards 16-byte FP32 optimizer states (Adam and ); reduces memory by with zero extra communication
- •ZeRO-2: shards gradients; ranks reduce-scatter gradients and retain only their partition; reduces memory by
- •ZeRO-3 (FSDP): shards model parameters; gathers full weights on-the-fly via All-Gather before each layer forward/backward, and discards them immediately
📐 Mathematical Derivations
Memory Breakdown and Sharding Mechanics (Rajbhandari et al., DeepSpeed, SC 2020):
For a model with parameters trained in FP16 mixed-precision with Adam:
- Model Weights (FP16): bytes.
- Gradients (FP16): bytes.
- Optimizer States (FP32 master weights , momentum , variance ): bytes.
Total baseline memory per GPU: bytes (e.g., 256GB for a 16B parameter model).
ZeRO Stages across $N$ Data-Parallel Ranks:
① ZeRO-1 ($P_{\text{os}}$): Shards optimizer states across ranks. Memory becomes . Communication: exact same All-Reduce overhead as standard DDP.
② ZeRO-2 ($P_{\text{os+g}}$): Shards both optimizer states and gradients. Gradients are synchronized via Reduce-Scatter instead of All-Reduce. Each rank updates only its parameter partition. Memory: . Communication: identical volume to standard DDP ().
③ ZeRO-3 ($P_{\text{os+g+p}}$): Shards model parameters as well. Each rank stores only parameters. Memory drops to bytes. Prior to each layer's forward pass, ranks perform an All-Gather to reconstruct parameters, compute layer, and release weights. Backward pass executes a second All-Gather. Increases communication volume by .
For a model with parameters trained in FP16 mixed-precision with Adam:
- Model Weights (FP16): bytes.
- Gradients (FP16): bytes.
- Optimizer States (FP32 master weights , momentum , variance ): bytes.
Total baseline memory per GPU: bytes (e.g., 256GB for a 16B parameter model).
ZeRO Stages across $N$ Data-Parallel Ranks:
① ZeRO-1 ($P_{\text{os}}$): Shards optimizer states across ranks. Memory becomes . Communication: exact same All-Reduce overhead as standard DDP.
② ZeRO-2 ($P_{\text{os+g}}$): Shards both optimizer states and gradients. Gradients are synchronized via Reduce-Scatter instead of All-Reduce. Each rank updates only its parameter partition. Memory: . Communication: identical volume to standard DDP ().
③ ZeRO-3 ($P_{\text{os+g+p}}$): Shards model parameters as well. Each rank stores only parameters. Memory drops to bytes. Prior to each layer's forward pass, ranks perform an All-Gather to reconstruct parameters, compute layer, and release weights. Backward pass executes a second All-Gather. Increases communication volume by .
🏭 Production Trade-offs
System design trade-offs: ZeRO-1/2 provide substantial memory relief with zero additional network traffic. ZeRO-3 allows training multi-billion parameter models without Tensor Parallelism, but incurs 50% more communication, requiring high-bandwidth interconnects (InfiniBand/RoCE).
⚠️ Common Interview Pitfalls
- ✕Using ZeRO-3 on clusters with slow 10Gbps Ethernet, causing training to be 90% communication-bound
- ✕Assuming ZeRO-1 increases network communication volume; ZeRO-1 has mathematically identical communication to PyTorch DDP
🎯 Interviewer Follow-up Inquiries
- ?Why does ZeRO-3 require 50% more communication volume than standard Data Parallelism (DDP)?
- ?How does ZeRO-Offload leverage host CPU RAM and NVMe SSDs to train massive models on single GPUs?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚