🧠 Deep LearningID: weight-decay

Weight Decay & AdamW

Q: Derive the equivalence of L2 regularization and weight decay under plain SGD: write both updates $\theta \leftarrow \theta - \eta(\nabla L+\lambda\theta)$ and $\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L$ and state the equivalence condition.

【Weight Decay & AdamW 核心考点 1】Derive the equivalence of L2 regularization and weight decay under plain SGD: write both updates $\theta \leftarrow \theta - \eta(\nabla L+\lambda\theta)$ and $\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L$ and state the equivalence condition.。技术解析：Definition: L2 regularization adds \frac{\lambda}{2}\Vert\theta\Vert_2^2 to the loss, turning the update into \theta \leftarrow \theta - \eta(\nabla L + \lambda\theta); under plain SGD this is exactly equivalent to weight decay \theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/weight-decay

Q: Why are L2 and weight decay inequivalent under Adam? What happens when $\lambda\theta$ is divided by $\sqrt{\hat{v}}+\epsilon$?

【Weight Decay & AdamW 核心考点 2】Why are L2 and weight decay inequivalent under Adam? What happens when $\lambda\theta$ is divided by $\sqrt{\hat{v}}+\epsilon$?。技术解析：Definition: L2 regularization adds \frac{\lambda}{2}\Vert\theta\Vert_2^2 to the loss, turning the update into \theta \leftarrow \theta - \eta(\nabla L + \lambda\theta); under plain SGD this is exactly equivalent to weight decay \theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/weight-decay

Q: Write the decoupled AdamW update $\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}$ and explain why it generalizes better than Adam + L2.

【Weight Decay & AdamW 核心考点 3】Write the decoupled AdamW update $\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}$ and explain why it generalizes better than Adam + L2.。技术解析：Definition: L2 regularization adds \frac{\lambda}{2}\Vert\theta\Vert_2^2 to the loss, turning the update into \theta \leftarrow \theta - \eta(\nabla L + \lambda\theta); under plain SGD this is exactly equivalent to weight decay \theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/weight-decay

Q: Hyperparameter scale: what is typical for AdamW weight decay (0.01–0.1)? How does it relate to L2's common 5e-4?

【Weight Decay & AdamW 核心考点 4】Hyperparameter scale: what is typical for AdamW weight decay (0.01–0.1)? How does it relate to L2's common 5e-4?。技术解析：Definition: L2 regularization adds \frac{\lambda}{2}\Vert\theta\Vert_2^2 to the loss, turning the update into \theta \leftarrow \theta - \eta(\nabla L + \lambda\theta); under plain SGD this is exactly equivalent to weight decay \theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/weight-decay

Q: Does weight decay drive weights all the way to zero? Why does it mostly constrain the norm rather than zero out parameters?

【Weight Decay & AdamW 核心考点 5】Does weight decay drive weights all the way to zero? Why does it mostly constrain the norm rather than zero out parameters?。技术解析：Definition: L2 regularization adds \frac{\lambda}{2}\Vert\theta\Vert_2^2 to the loss, turning the update into \theta \leftarrow \theta - \eta(\nabla L + \lambda\theta); under plain SGD this is exactly equivalent to weight decay \theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/weight-decay

权重衰减与 AdamW

🎯Core Definition

L2 regularization adds

\frac{\lambda}{2}\Vert\theta\Vert_2^2

to the loss, turning the update into

\theta \leftarrow \theta - \eta(\nabla L + \lambda\theta)

; under plain SGD this is exactly equivalent to weight decay

\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L

. With Adam, however, the gradient term is normalized by

\sqrt{\hat{v}} + \epsilon

, which divides

\lambda\theta

by the same factor, so the effective decay varies wildly with gradient magnitude — L2 and weight decay are no longer equivalent. AdamW decouples them by applying weight decay directly to the parameters:

\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}

💡Use Cases

the standard generalization tool for deep networks under Adam/AdamW training, especially LLM pretraining and finetuning; interviews ask for the SGD equivalence derivation, why it breaks under Adam, and AdamW's improvement.

⚡Key Problems Solved

constrains the parameter norm to prevent overfitting. AdamW (Loshchilov & Hutter 2019) fixes the mis-scaled weight decay under Adam: the decay coefficient is constant and independent of gradient magnitude, improving generalization (the paper reports roughly 2% lower test error) and robustness to learning-rate / weight-decay hyperparameters; LLM pretraining (e.g. LLaMA) standardly uses AdamW with weight decay around 0.01–0.1 (an order of magnitude above naive L2's 5e-4).

🎯5 High-Frequency Exam Points

Derive the equivalence of L2 regularization and weight decay under plain SGD: write both updates

\theta \leftarrow \theta - \eta(\nabla L+\lambda\theta)

and

\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L

and state the equivalence condition.

Why are L2 and weight decay inequivalent under Adam? What happens when

\lambda\theta

is divided by

\sqrt{\hat{v}}+\epsilon

Write the decoupled AdamW update

\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}

and explain why it generalizes better than Adam + L2.

Hyperparameter scale: what is typical for AdamW weight decay (0.01–0.1)? How does it relate to L2's common 5e-4?

Does weight decay drive weights all the way to zero? Why does it mostly constrain the norm rather than zero out parameters?

📖 In-depth Guide:📄 normalization-and-regularization →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Weight Decay & AdamW"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardLabel Smoothing Next Card →Convolution Basics

🔗 More Deep Learning Knowledge Cards

Activation Functions Adam & AdamW Autograd Compute Graph Batch Normalization