H1Hard★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems

Adam & AdamW Optimizer from Scratch

⚡Flashcard Drill➔

Implement Adam first/second moment bias correction alongside decoupled AdamW weight decay.

⏱️ Time: O(P) 每个参数常数次更新
💾 Space: 显存占用为参数量的 2 倍(需持久保存 m 和 v 状态)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class AdamW:
def __init__(self, params: list, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01):
self.params = params
self.lr = lr
self.beta1, self.beta2 = betas
self.eps = eps
self.weight_decay = weight_decay
self.t = 0
# 初始化动量与方差缓存
self.m = [np.zeros_like(p) for p in params]
self.v = [np.zeros_like(p) for p in params]
def step(self, grads: list):
self.t += 1
for i, (p, g) in enumerate(zip(self.params, grads)):
# 1. AdamW 解耦权重衰减: 独立直接衰减当前参数
if self.weight_decay != 0.0:
p -= self.lr * self.weight_decay * p
# 2. 一阶与二阶矩更新
self.m[i] =
self.v[i] = self.beta2 * self.v[i] + (1.0 - self.beta2) * (g ** 2)
# 3. 偏差校正 (Bias correction)
m_hat = self.m[i] / (1.0 - self.beta1 ** self.t)
v_hat =
# 4. 参数步进更新
p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does ZeRO-1 reduce Adam optimizer state memory across distributed GPUs?
Next: Learning Rate Scheduler: Linear Warmup & Cosine Decay →
💡

Core Mental Anchor / Mnemonic

Track bias-corrected m_hat and v_hat, then decouple weight decay directly on parameters.

📐 Mathematical Derivation & Core Formula

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,θt=θt−1−ηλθt−1−ηv^t+ϵm^tm_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2, \quad \theta_t = \theta_{t-1} - \eta \lambda \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
### Adam Bias Correction & AdamW Decoupled Weight Decay
1. Adam: Tracks exponential moving averages of gradients (mtm_t) and squared gradients (vtv_t). Dividing by (1−β1t)(1 - \beta_1^t) and (1−β2t)(1 - \beta_2^t) corrects initial zero-initialization bias at small tt;
2. AdamW: Coupling L2 decay inside the gradient (gt+λθg_t + \lambda \theta) scales regularization by 1/vt1/\sqrt{v_t}, distorting effective decay across parameters. Decoupled AdamW subtracts ηλθ\eta \lambda \theta directly during the parameter update.

🔄 Tensor Dimensions & Shape Flow

g -> update m, v -> bias-correct m_hat, v_hat -> decoupled decay p -= lr*wd*p -> p -= lr*m_hat/(sqrt(v_hat)+eps)

🛡️ Numerical Stability & Pitfalls Checklist

  • Add eps outside the square root (sqrt(v_hat) + eps) as in standard PyTorch AdamW.
  • LLM pretraining commonly sets beta2 = 0.95 instead of 0.999 to react faster to gradient spikes.
  • Exclude LayerNorm gains/biases and linear biases from weight decay.

💻 Reference Code Implementation

import numpy as np

class AdamW:
    def __init__(self, params: list, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01):
        self.params = params
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.eps = eps
        self.weight_decay = weight_decay
        self.t = 0
        
        # 初始化动量与方差缓存
        self.m = [np.zeros_like(p) for p in params]
        self.v = [np.zeros_like(p) for p in params]
        
    def step(self, grads: list):
        self.t += 1
        for i, (p, g) in enumerate(zip(self.params, grads)):
            # 1. AdamW 解耦权重衰减: 独立直接衰减当前参数
            if self.weight_decay != 0.0:
                p -= self.lr * self.weight_decay * p
                
            # 2. 一阶与二阶矩更新
            self.m[i] = self.beta1 * self.m[i] + (1.0 - self.beta1) * g
            self.v[i] = self.beta2 * self.v[i] + (1.0 - self.beta2) * (g ** 2)
            
            # 3. 偏差校正 (Bias correction)
            m_hat = self.m[i] / (1.0 - self.beta1 ** self.t)
            v_hat = self.v[i] / (1.0 - self.beta2 ** self.t)
            
            # 4. 参数步进更新
            p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
w = np.array([5.0])
opt = AdamW([w], lr=0.1, weight_decay=0.01)
# 模拟恒定正梯度 1.0
opt.step([np.array([1.0])])
# 第一步 w 应该减小
assert w[0] < 5.0
print("✓ AdamW 优化器自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does ZeRO-1 reduce Adam optimizer state memory across distributed GPUs?
Adam keeps FP32 m and v states totaling 8Φ8\Phi bytes for Φ\Phi parameters. ZeRO-1 shards these optimizer states evenly across NN data-parallel ranks so each GPU holds only 1/N1/N of the states without increasing total communication volume.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Vision Transformer (ViT) Patch EmbeddingBack to Part H & All 105Next: Learning Rate Scheduler: Linear Warmup & Cosine Decay →