M3-031M3: Deep Learning FoundationsOptimizers & Second-Order MethodsEasy
⚡Flashcard Drill➔Mastery:

Adam vs AdamW: Structural Differences and Why AdamW is Mathematically Correct

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-032 →
📐 Mathematical Definition
Adam: g←∇f+λθ;AdamW: θ←θ(1−ηλ)−ηm^v^+ϵ\text{Adam}:\ g\leftarrow\nabla f+\lambda\theta;\qquad \text{AdamW}:\ \theta\leftarrow\theta(1-\eta\lambda)-\eta\frac{\hat m}{\sqrt{\hat v}+\epsilon}
⚡ One-Sentence Core Answer
Adam couples L2 regularization with adaptive gradient moments, dampening decay for high-gradient parameters; AdamW restores true weight decay by decoupling it from gradient normalization.

📌 Key Takeaways

  • •
    Adam flaw: adds λθ\lambda \theta directly to gradient gtg_t, dividing the penalty by vt\sqrt{v_t} and distorting regularization
  • •
    AdamW fix: decays weights directly: θt+1=(1−ηλ)θt−ηv^t+ϵm^t\theta_{t+1} = (1 - \eta \lambda) \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
  • •
    Separation of concerns: AdamW decouples weight regularization from gradient scale and optimizer step variance

📐 Mathematical Derivations

Mathematical Comparison (Loshchilov & Hutter, ICLR 2019):
- Standard Adam with L2 Regularization:
Loss objective: L~(θ)=L(θ)+λ2∥θ∥2\tilde{\mathcal{L}}(\theta) = \mathcal{L}(\theta) + \frac{\lambda}{2} \|\theta\|^2. The gradient is gt′=∇L(θ)+λθtg_t' = \nabla \mathcal{L}(\theta) + \lambda \theta_t.
The update rule computes moments on gt′g_t': vt=β2vt−1+(1−β2)(gt+λθt)2v_t = \beta_2 v_{t-1} + (1 - \beta_2) (g_t + \lambda \theta_t)^2.
The resulting parameter update is: Δθt=−ηvt+ϵ(gt+λθt)=−ηvt+ϵgt−ηλvt+ϵθt\Delta \theta_t = - \frac{\eta}{\sqrt{v_t} + \epsilon} (g_t + \lambda \theta_t) = - \frac{\eta}{\sqrt{v_t} + \epsilon} g_t - \frac{\eta \lambda}{\sqrt{v_t} + \epsilon} \theta_t.
The Defect: For parameters with large historical gradients (vt≫1v_t \gg 1), effective weight decay ηλvt\frac{\eta \lambda}{\sqrt{v_t}} is suppressed! Parameters with tiny gradients undergo excessive decay.
- Adam with Decoupled Weight Decay (AdamW):
Moments are computed strictly on clean gradient gt=∇L(θ)g_t = \nabla \mathcal{L}(\theta).
Update rule: θt+1=θt−ηλθt−ηv^t+ϵm^t=(1−ηλ)θt−ηv^t+ϵm^t\theta_{t+1} = \theta_t - \eta \lambda \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t = (1 - \eta \lambda) \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t.
Every parameter decays at constant rate ηλ\eta \lambda, restoring true scale-invariant regularization.

🏭 Production Trade-offs

Standard adoption: AdamW is universally mandated across all Transformer pre-training (BERT, GPT-3, LLaMA) and computer vision foundation models. Standard Adam with L2 is considered deprecated for deep architectures.
⚠️ Common Interview Pitfalls
  • ✕
    Using PyTorch torch.optim.Adam(weight_decay=0.01) expecting AdamW behavior; you must use torch.optim.AdamW
  • ✕
    Applying decoupled weight decay to 1D parameters (LayerNorm gains γ\gamma, biases bb), which impairs normalization stability
🎯 Interviewer Follow-up Inquiries
  • ?
    Why should weight decay be excluded for bias vectors and normalization layer scale parameters?
  • ?
    Under standard SGD, are L2 regularization and decoupled weight decay identical? Prove why.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-030: Update Formulas for SGD, SGD with Momentum, and Adam📋 Back to BankNext →M3-032: Role of the Epsilon Parameter in Adam and Common Settings in Large Models