H8Medium★ Distributed Systems Essential · Memory BudgetingPart H · Optimizers & Distributed Systems
ZeRO-1/2/3 Model-State Memory Calculator
Quantify per-GPU model-state memory across parameters (2B), gradients (2B), and FP32 Adam states (12B) under ZeRO stages 0–3.
⏱️ Time:
O(1) 显存分析模型💾 Space:
O(1)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def zero_memory_per_gpu_bytes(num_params: int, dp_world_size: int, stage: int) -> float:
"""
Compute steady-state model-state memory (bytes) per GPU under mixed-precision Adam.
Baseline (total = 16 bytes/param): FP16 params (2B) + FP16 grads (2B) + FP32 optimizer states
(master weights 4B + Adam m 4B + Adam v 4B = 12B).
"""
if dp_world_size < 1:
raise ValueError("dp_world_size must be >= 1")
params_b =
grads_b = 2.0 * num_params
opt_b = 12.0 * num_params
if stage == 0:
return params_b + grads_b + opt_b
if stage == 1:
return params_b + grads_b + (opt_b / dp_world_size)
if stage == 2:
return params_b + (grads_b + opt_b) / dp_world_size
if stage == 3:
return
raise ValueError(f"Unsupported ZeRO stage: {stage}")
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How much additional communication overhead does ZeRO-3 introduce compared to ZeRO-2?
💡
Core Mental Anchor / Mnemonic
2B weights + 2B grads + 12B FP32 Adam = 16B total; ZeRO-1 shards 12B, ZeRO-2 shards 14B, ZeRO-3 shards all 16B.
📐 Mathematical Derivation & Core Formula
### The 16-Byte-Per-Parameter Budget
Mixed-precision Adam maintains (FP16 weights) + (FP16 gradients) + (FP32 master weights, momentum , variance ) = bytes of model states across data-parallel GPUs:
- ZeRO-1: Shards optimizer states () with zero extra communication;
- ZeRO-2: Also shards gradients ();
- ZeRO-3 (FSDP): Shards all three (), trading ~1.5x total communication volume (
Mixed-precision Adam maintains (FP16 weights) + (FP16 gradients) + (FP32 master weights, momentum , variance ) = bytes of model states across data-parallel GPUs:
- ZeRO-1: Shards optimizer states () with zero extra communication;
- ZeRO-2: Also shards gradients ();
- ZeRO-3 (FSDP): Shards all three (), trading ~1.5x total communication volume (
All-Gather weights before forward/backward) for linear memory reduction.🔄 Tensor Dimensions & Shape Flow
num_params, dp_world_size, stage -> shard 12B opt / 2B grad / 2B param -> bytes per GPU
🛡️ Numerical Stability & Pitfalls Checklist
- Always budget activation memory and collective scratch buffers on top of steady-state model-state bytes.
- Prefer ZeRO-1/2 whenever parameters fit cleanly to avoid ZeRO-3's extra forward parameter All-Gather overhead.
💻 Reference Code Implementation
def zero_memory_per_gpu_bytes(num_params: int, dp_world_size: int, stage: int) -> float:
"""
Compute steady-state model-state memory (bytes) per GPU under mixed-precision Adam.
Baseline (total = 16 bytes/param): FP16 params (2B) + FP16 grads (2B) + FP32 optimizer states
(master weights 4B + Adam m 4B + Adam v 4B = 12B).
"""
if dp_world_size < 1:
raise ValueError("dp_world_size must be >= 1")
params_b = 2.0 * num_params
grads_b = 2.0 * num_params
opt_b = 12.0 * num_params
if stage == 0:
return params_b + grads_b + opt_b
if stage == 1:
return params_b + grads_b + (opt_b / dp_world_size)
if stage == 2:
return params_b + (grads_b + opt_b) / dp_world_size
if stage == 3:
return (params_b + grads_b + opt_b) / dp_world_size
raise ValueError(f"Unsupported ZeRO stage: {stage}")
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
psi = 7_000_000_000 # 7B model
N = 8
m0 = zero_memory_per_gpu_bytes(psi, N, 0)
m1 = zero_memory_per_gpu_bytes(psi, N, 1)
m2 = zero_memory_per_gpu_bytes(psi, N, 2)
m3 = zero_memory_per_gpu_bytes(psi, N, 3)
assert m0 > m1 > m2 > m3, "Memory per rank must decrease monotonically with ZeRO stage"
assert abs(m0 - 16.0 * psi) < 1e-6 and abs(m3 - 2.0 * psi) < 1e-6
print("✓ ZeRO-1/2/3 memory footprint assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How much additional communication overhead does ZeRO-3 introduce compared to ZeRO-2?
ZeRO-2 incurs ~2*Psi gradient communication per step (Reduce-Scatter + All-Gather), whereas ZeRO-3 adds weight All-Gather before forward and backward (~3*Psi total, ~50% higher collective volume).
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.