M3-048M3: Deep Learning FoundationsRegularization & Training TricksEasy
⚡Flashcard Drill➔Mastery:
Why Weight Decay is Decoupled from Learning Rate
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Coupled L2 regularization distorts decay rates when scaled by adaptive second moments; decoupling applies uniform decay directly to parameters.
📌 Key Takeaways
- •Coupled defect: gradient modification causes effective decay to vary inversely with gradient magnitude
- •Decoupled formulation:
- •Consistency across schedules: when learning rate anneals, the effective shrinkage rate anneals proportionally
📐 Mathematical Derivations
Mathematical Comparison:
In standard SGD: .
Update: . In SGD, L2 regularization and decoupled weight decay are mathematically identical.
In adaptive optimizers (Adam): Update uses adaptive preconditioner .
- Coupled L2: . The decay rate is scaled by , punishing low-gradient parameters heavily while ignoring high-gradient weights.
- Decoupled Decay (AdamW): . The decay rate is independent of gradient history.
In standard SGD: .
Update: . In SGD, L2 regularization and decoupled weight decay are mathematically identical.
In adaptive optimizers (Adam): Update uses adaptive preconditioner .
- Coupled L2: . The decay rate is scaled by , punishing low-gradient parameters heavily while ignoring high-gradient weights.
- Decoupled Decay (AdamW): . The decay rate is independent of gradient history.
🏭 Production Trade-offs
Parameter specificity: In Transformer models, apply weight decay strictly to 2D/3D weight matrices. Never apply weight decay to 1D biases or LayerNorm/RMSNorm scale parameters ().
⚠️ Common Interview Pitfalls
- ✕Applying weight decay to bias vectors, which introduces an arbitrary shrinkage prior on constant baseline offsets
- ✕Assuming changing the learning rate schedule has no effect on weight decay; decoupled decay scales multiplicatively with
🎯 Interviewer Follow-up Inquiries
- ?How do you configure PyTorch parameter groups to apply weight decay only to weights and not to biases or LayerNorms?
- ?Why is decoupled weight decay mathematically identical to L2 regularization in momentum SGD?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚