M3-048M3: Deep Learning FoundationsRegularization & Training TricksEasy
⚡Flashcard Drill➔Mastery:

Why Weight Decay is Decoupled from Learning Rate

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-049 →
📐 Mathematical Definition
θ←θ−ηλθ−ηm^v^+ϵ\theta\leftarrow\theta-\eta\lambda\theta-\eta\frac{\hat m}{\sqrt{\hat v}+\epsilon}
⚡ One-Sentence Core Answer
Coupled L2 regularization distorts decay rates when scaled by adaptive second moments; decoupling applies uniform decay (1−ηλ)(1 - \eta \lambda) directly to parameters.

📌 Key Takeaways

  • •
    Coupled defect: gradient modification g+λθg + \lambda \theta causes effective decay to vary inversely with gradient magnitude
  • •
    Decoupled formulation: θt+1=(1−ηλ)θt−η⋅OptimizerStep(gt)\theta_{t+1} = (1 - \eta \lambda) \theta_t - \eta \cdot \text{OptimizerStep}(g_t)
  • •
    Consistency across schedules: when learning rate ηt\eta_t anneals, the effective shrinkage rate ηtλ\eta_t \lambda anneals proportionally

📐 Mathematical Derivations

Mathematical Comparison:
In standard SGD: Lreg=L+λ2∥θ∥2  ⟹  g=∇L+λθ\mathcal{L}_{\text{reg}} = \mathcal{L} + \frac{\lambda}{2}\|\theta\|^2 \implies g = \nabla \mathcal{L} + \lambda \theta.
Update: θt+1=θt−η(∇L+λθt)=(1−ηλ)θt−η∇L\theta_{t+1} = \theta_t - \eta (\nabla \mathcal{L} + \lambda \theta_t) = (1 - \eta \lambda) \theta_t - \eta \nabla \mathcal{L}. In SGD, L2 regularization and decoupled weight decay are mathematically identical.
In adaptive optimizers (Adam): Update uses adaptive preconditioner Dt=diag(vt+ϵ)−1D_t = \text{diag}(\sqrt{v_t} + \epsilon)^{-1}.
- Coupled L2: θt+1=θt−ηDt(∇L+λθt)=θt−ηDt∇L−ηλDtθt\theta_{t+1} = \theta_t - \eta D_t (\nabla \mathcal{L} + \lambda \theta_t) = \theta_t - \eta D_t \nabla \mathcal{L} - \eta \lambda D_t \theta_t. The decay rate is scaled by DtD_t, punishing low-gradient parameters heavily while ignoring high-gradient weights.
- Decoupled Decay (AdamW): θt+1=θt−ηDt∇L−ηλθt=(1−ηλ)θt−ηDt∇L\theta_{t+1} = \theta_t - \eta D_t \nabla \mathcal{L} - \eta \lambda \theta_t = (1 - \eta \lambda) \theta_t - \eta D_t \nabla \mathcal{L}. The decay rate ηλ\eta \lambda is independent of gradient history.

🏭 Production Trade-offs

Parameter specificity: In Transformer models, apply weight decay strictly to 2D/3D weight matrices. Never apply weight decay to 1D biases or LayerNorm/RMSNorm scale parameters (γ\gamma).
⚠️ Common Interview Pitfalls
  • ✕
    Applying weight decay to bias vectors, which introduces an arbitrary shrinkage prior on constant baseline offsets
  • ✕
    Assuming changing the learning rate schedule has no effect on weight decay; decoupled decay scales multiplicatively with ηt\eta_t
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you configure PyTorch parameter groups to apply weight decay only to weights and not to biases or LayerNorms?
  • ?
    Why is decoupled weight decay mathematically identical to L2 regularization in momentum SGD?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-047: Major Regularization Techniques in Deep Learning📋 Back to BankNext →M3-049: Mixup and CutMix Data Augmentations: Mechanisms and Theoretical Foundations