H8Medium★ Distributed Systems Essential · Memory BudgetingPart H · Optimizers & Distributed Systems

ZeRO-1/2/3 Model-State Memory Calculator

⚡Flashcard Drill➔

Quantify per-GPU model-state memory across parameters (2B), gradients (2B), and FP32 Adam states (12B) under ZeRO stages 0–3.

⏱️ Time: O(1) 显存分析模型
💾 Space: O(1)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def zero_memory_per_gpu_bytes(num_params: int, dp_world_size: int, stage: int) -> float:
"""
Compute steady-state model-state memory (bytes) per GPU under mixed-precision Adam.
Baseline (total = 16 bytes/param): FP16 params (2B) + FP16 grads (2B) + FP32 optimizer states
(master weights 4B + Adam m 4B + Adam v 4B = 12B).
"""
if dp_world_size < 1:
raise ValueError("dp_world_size must be >= 1")
params_b =
grads_b = 2.0 * num_params
opt_b = 12.0 * num_params
if stage == 0:
return params_b + grads_b + opt_b
if stage == 1:
return params_b + grads_b + (opt_b / dp_world_size)
if stage == 2:
return params_b + (grads_b + opt_b) / dp_world_size
if stage == 3:
return
raise ValueError(f"Unsupported ZeRO stage: {stage}")
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How much additional communication overhead does ZeRO-3 introduce compared to ZeRO-2?
Next: INT8 Affine & Symmetric Quantization →
💡

Core Mental Anchor / Mnemonic

2B weights + 2B grads + 12B FP32 Adam = 16B total; ZeRO-1 shards 12B, ZeRO-2 shards 14B, ZeRO-3 shards all 16B.

📐 Mathematical Derivation & Core Formula

MemZeRO−1=2Ψ+2Ψ+12ΨN,MemZeRO−3=16ΨN\mathrm{Mem}_{\mathrm{ZeRO}-1} = 2\Psi + 2\Psi + \frac{12\Psi}{N}, \quad \mathrm{Mem}_{\mathrm{ZeRO}-3} = \frac{16\Psi}{N}
### The 16-Byte-Per-Parameter Budget
Mixed-precision Adam maintains 2Ψ2\Psi (FP16 weights) + 2Ψ2\Psi (FP16 gradients) + 12Ψ12\Psi (FP32 master weights, momentum mm, variance vv) = 16Ψ16\Psi bytes of model states across NN data-parallel GPUs:
- ZeRO-1: Shards optimizer states (2Ψ+2Ψ+12Ψ/N2\Psi + 2\Psi + 12\Psi/N) with zero extra communication;
- ZeRO-2: Also shards gradients (2Ψ+14Ψ/N2\Psi + 14\Psi/N);
- ZeRO-3 (FSDP): Shards all three (16Ψ/N16\Psi/N), trading ~1.5x total communication volume (All-Gather weights before forward/backward) for linear memory reduction.

🔄 Tensor Dimensions & Shape Flow

num_params, dp_world_size, stage -> shard 12B opt / 2B grad / 2B param -> bytes per GPU

🛡️ Numerical Stability & Pitfalls Checklist

  • Always budget activation memory and collective scratch buffers on top of steady-state model-state bytes.
  • Prefer ZeRO-1/2 whenever parameters fit cleanly to avoid ZeRO-3's extra forward parameter All-Gather overhead.

💻 Reference Code Implementation

def zero_memory_per_gpu_bytes(num_params: int, dp_world_size: int, stage: int) -> float:
    """
    Compute steady-state model-state memory (bytes) per GPU under mixed-precision Adam.
    Baseline (total = 16 bytes/param): FP16 params (2B) + FP16 grads (2B) + FP32 optimizer states
    (master weights 4B + Adam m 4B + Adam v 4B = 12B).
    """
    if dp_world_size < 1:
        raise ValueError("dp_world_size must be >= 1")
    params_b = 2.0 * num_params
    grads_b = 2.0 * num_params
    opt_b = 12.0 * num_params
    if stage == 0:
        return params_b + grads_b + opt_b
    if stage == 1:
        return params_b + grads_b + (opt_b / dp_world_size)
    if stage == 2:
        return params_b + (grads_b + opt_b) / dp_world_size
    if stage == 3:
        return (params_b + grads_b + opt_b) / dp_world_size
    raise ValueError(f"Unsupported ZeRO stage: {stage}")

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

psi = 7_000_000_000  # 7B model
N = 8
m0 = zero_memory_per_gpu_bytes(psi, N, 0)
m1 = zero_memory_per_gpu_bytes(psi, N, 1)
m2 = zero_memory_per_gpu_bytes(psi, N, 2)
m3 = zero_memory_per_gpu_bytes(psi, N, 3)
assert m0 > m1 > m2 > m3, "Memory per rank must decrease monotonically with ZeRO stage"
assert abs(m0 - 16.0 * psi) < 1e-6 and abs(m3 - 2.0 * psi) < 1e-6
print("✓ ZeRO-1/2/3 memory footprint assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How much additional communication overhead does ZeRO-3 introduce compared to ZeRO-2?
ZeRO-2 incurs ~2*Psi gradient communication per step (Reduce-Scatter + All-Gather), whereas ZeRO-3 adds weight All-Gather before forward and backward (~3*Psi total, ~50% higher collective volume).
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Megatron Tensor Parallelism (Column & Row Linear)Back to Part H & All 105Next: INT8 Affine & Symmetric Quantization →