H1Hard★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems
Adam & AdamW Optimizer from Scratch
Implement Adam first/second moment bias correction alongside decoupled AdamW weight decay.
⏱️ Time:
O(P) 每个参数常数次更新💾 Space:
显存占用为参数量的 2 倍(需持久保存 m 和 v 状态)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class AdamW:
def __init__(self, params: list, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01):
self.params = params
self.lr = lr
self.beta1, self.beta2 = betas
self.eps = eps
self.weight_decay = weight_decay
self.t = 0
# 初始化动量与方差缓存
self.m = [np.zeros_like(p) for p in params]
self.v = [np.zeros_like(p) for p in params]
def step(self, grads: list):
self.t += 1
for i, (p, g) in enumerate(zip(self.params, grads)):
# 1. AdamW 解耦权重衰减: 独立直接衰减当前参数
if self.weight_decay != 0.0:
p -= self.lr * self.weight_decay * p
# 2. 一阶与二阶矩更新
self.m[i] =
self.v[i] = self.beta2 * self.v[i] + (1.0 - self.beta2) * (g ** 2)
# 3. 偏差校正 (Bias correction)
m_hat = self.m[i] / (1.0 - self.beta1 ** self.t)
v_hat =
# 4. 参数步进更新
p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does ZeRO-1 reduce Adam optimizer state memory across distributed GPUs?
💡
Core Mental Anchor / Mnemonic
Track bias-corrected m_hat and v_hat, then decouple weight decay directly on parameters.
📐 Mathematical Derivation & Core Formula
### Adam Bias Correction & AdamW Decoupled Weight Decay
1. Adam: Tracks exponential moving averages of gradients () and squared gradients (). Dividing by and corrects initial zero-initialization bias at small ;
2. AdamW: Coupling L2 decay inside the gradient () scales regularization by , distorting effective decay across parameters. Decoupled AdamW subtracts directly during the parameter update.
1. Adam: Tracks exponential moving averages of gradients () and squared gradients (). Dividing by and corrects initial zero-initialization bias at small ;
2. AdamW: Coupling L2 decay inside the gradient () scales regularization by , distorting effective decay across parameters. Decoupled AdamW subtracts directly during the parameter update.
🔄 Tensor Dimensions & Shape Flow
g -> update m, v -> bias-correct m_hat, v_hat -> decoupled decay p -= lr*wd*p -> p -= lr*m_hat/(sqrt(v_hat)+eps)
🛡️ Numerical Stability & Pitfalls Checklist
- Add
epsoutside the square root (sqrt(v_hat) + eps) as in standard PyTorch AdamW. - LLM pretraining commonly sets
beta2 = 0.95instead of0.999to react faster to gradient spikes. - Exclude LayerNorm gains/biases and linear biases from weight decay.
💻 Reference Code Implementation
import numpy as np
class AdamW:
def __init__(self, params: list, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01):
self.params = params
self.lr = lr
self.beta1, self.beta2 = betas
self.eps = eps
self.weight_decay = weight_decay
self.t = 0
# 初始化动量与方差缓存
self.m = [np.zeros_like(p) for p in params]
self.v = [np.zeros_like(p) for p in params]
def step(self, grads: list):
self.t += 1
for i, (p, g) in enumerate(zip(self.params, grads)):
# 1. AdamW 解耦权重衰减: 独立直接衰减当前参数
if self.weight_decay != 0.0:
p -= self.lr * self.weight_decay * p
# 2. 一阶与二阶矩更新
self.m[i] = self.beta1 * self.m[i] + (1.0 - self.beta1) * g
self.v[i] = self.beta2 * self.v[i] + (1.0 - self.beta2) * (g ** 2)
# 3. 偏差校正 (Bias correction)
m_hat = self.m[i] / (1.0 - self.beta1 ** self.t)
v_hat = self.v[i] / (1.0 - self.beta2 ** self.t)
# 4. 参数步进更新
p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
w = np.array([5.0])
opt = AdamW([w], lr=0.1, weight_decay=0.01)
# 模拟恒定正梯度 1.0
opt.step([np.array([1.0])])
# 第一步 w 应该减小
assert w[0] < 5.0
print("✓ AdamW 优化器自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does ZeRO-1 reduce Adam optimizer state memory across distributed GPUs?
Adam keeps FP32
m and v states totaling bytes for parameters. ZeRO-1 shards these optimizer states evenly across data-parallel ranks so each GPU holds only of the states without increasing total communication volume.⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.