H1Hard★ Core Essential · Industrial BedrockPart H · Optimizers & Distributed Systems

Adam & AdamW Optimizer from Scratch

Mastery Status

Industrial-grade implementation and mathematical foundations of Adam & AdamW Optimizer from Scratch.

⏱️ Time Complexity: O(P) 每个参数常数次更新

💾 Space Complexity: 显存占用为参数量的 2 倍（需持久保存 m 和 v 状态）

💡

Core Mental Anchor / Mnemonic

Master Adam & AdamW Optimizer from Scratch: enforce numerical stability, check tensor shapes, and eliminate redundant memory allocations.

📐 Mathematical Derivation & Core Formula

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2, \quad \theta_t = \theta_{t-1} - \eta \lambda \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t

### Mathematical Derivation & Theoretical Principles
Detailed first-principles formulation and architectural mechanics for Adam & AdamW Optimizer from Scratch.

Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.

🔄 Tensor Dimensions & Shape Flow

g -> m, v 动量更新 -> m_hat, v_hat 偏差校正 -> p -= lr*wd*p -> p -= lr * m_hat / (sqrt(v_hat)+eps)

🛡️ Industrial Numerical Stability & Pitfalls

Ensure proper multi-dimensional tensor broadcasting and keepdims retention.
Enforce numerical guards (eps clamping and overflow thresholds) during exponentiation and division.
Verify train versus eval mode behavioral distinctions (e.g. frozen running statistics and dropout bypass).

💻 Industrial Code Implementation

import numpy as np

class AdamW:
    def __init__(self, params: list, lr: float = 1e-3, betas: tuple = (0.9, 0.999), eps: float = 1e-8, weight_decay: float = 0.01):
        self.params = params
        self.lr = lr
        self.beta1, self.beta2 = betas
        self.eps = eps
        self.weight_decay = weight_decay
        self.t = 0
        
        # 初始化动量与方差缓存
        self.m = [np.zeros_like(p) for p in params]
        self.v = [np.zeros_like(p) for p in params]
        
    def step(self, grads: list):
        self.t += 1
        for i, (p, g) in enumerate(zip(self.params, grads)):
            # 1. AdamW 解耦权重衰减: 独立直接衰减当前参数
            if self.weight_decay != 0.0:
                p -= self.lr * self.weight_decay * p
                
            # 2. 一阶与二阶矩更新
            self.m[i] = self.beta1 * self.m[i] + (1.0 - self.beta1) * g
            self.v[i] = self.beta2 * self.v[i] + (1.0 - self.beta2) * (g ** 2)
            
            # 3. 偏差校正 (Bias correction)
            m_hat = self.m[i] / (1.0 - self.beta1 ** self.t)
            v_hat = self.v[i] / (1.0 - self.beta2 ** self.t)
            
            # 4. 参数步进更新
            p -= self.lr * m_hat / (np.sqrt(v_hat) + self.eps)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter to verify correctness:

import numpy as np
w = np.array([5.0])
opt = AdamW([w], lr=0.1, weight_decay=0.01)
# 模拟恒定正梯度 1.0
opt.step([np.array([1.0])])
# 第一步 w 应该减小
assert w[0] < 5.0
print("✓ AdamW 优化器自测通过")

🎯 Core Architecture Follow-up Q&A

Q1:What are the key trade-offs and memory bottlenecks when deploying Adam & AdamW Optimizer from Scratch in high-throughput inference?

Memory bandwidth (HBM to SRAM I/O) is the primary latency factor. Fusing element-wise operations and avoiding intermediate tensor materialization significantly outperforms naive implementations.

Q2:How does Adam & AdamW Optimizer from Scratch handle extreme numerical boundaries or precision reduction (FP16/BF16/INT8)?

Under low precision, operations must be upcasted to FP32 during accumulation to prevent underflow/overflow, followed by proper scaling and clamping before converting back to the target format.

←Prev: Vision Transformer (ViT) Patch Embedding All 69 Kernels Next: Learning Rate Scheduler: Linear Warmup & Cosine Decay→