M3-031M3: Deep Learning FoundationsOptimizers & Second-Order MethodsEasy
⚡Flashcard Drill➔Mastery:
Adam vs AdamW: Structural Differences and Why AdamW is Mathematically Correct
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Adam couples L2 regularization with adaptive gradient moments, dampening decay for high-gradient parameters; AdamW restores true weight decay by decoupling it from gradient normalization.
📌 Key Takeaways
- •Adam flaw: adds directly to gradient , dividing the penalty by and distorting regularization
- •AdamW fix: decays weights directly:
- •Separation of concerns: AdamW decouples weight regularization from gradient scale and optimizer step variance
📐 Mathematical Derivations
Mathematical Comparison (Loshchilov & Hutter, ICLR 2019):
- Standard Adam with L2 Regularization:
Loss objective: . The gradient is .
The update rule computes moments on : .
The resulting parameter update is: .
The Defect: For parameters with large historical gradients (), effective weight decay is suppressed! Parameters with tiny gradients undergo excessive decay.
- Adam with Decoupled Weight Decay (AdamW):
Moments are computed strictly on clean gradient .
Update rule: .
Every parameter decays at constant rate , restoring true scale-invariant regularization.
- Standard Adam with L2 Regularization:
Loss objective: . The gradient is .
The update rule computes moments on : .
The resulting parameter update is: .
The Defect: For parameters with large historical gradients (), effective weight decay is suppressed! Parameters with tiny gradients undergo excessive decay.
- Adam with Decoupled Weight Decay (AdamW):
Moments are computed strictly on clean gradient .
Update rule: .
Every parameter decays at constant rate , restoring true scale-invariant regularization.
🏭 Production Trade-offs
Standard adoption: AdamW is universally mandated across all Transformer pre-training (BERT, GPT-3, LLaMA) and computer vision foundation models. Standard Adam with L2 is considered deprecated for deep architectures.
⚠️ Common Interview Pitfalls
- ✕Using PyTorch
torch.optim.Adam(weight_decay=0.01)expecting AdamW behavior; you must usetorch.optim.AdamW - ✕Applying decoupled weight decay to 1D parameters (LayerNorm gains , biases ), which impairs normalization stability
🎯 Interviewer Follow-up Inquiries
- ?Why should weight decay be excluded for bias vectors and normalization layer scale parameters?
- ?Under standard SGD, are L2 regularization and decoupled weight decay identical? Prove why.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚