Weight Decay & AdamW

L2 regularization adds $\frac{\lambda}{2}\Vert\theta\Vert_2^2$ to the loss, turning the update into $\theta \leftarrow \theta - \eta(\nabla L + \lambda\theta)$; under plain SGD this is exactly equivalent to weight decay $\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L$. With Adam, however, the gradient term is normalized by $\sqrt{\hat{v}} + \epsilon$, which divides $\lambda\theta$ by the same factor, so the effective decay varies wildly with gradient magnitude — L2 and weight decay are no longer equivalent. AdamW decouples them by applying weight decay directly to the parameters: $\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}$.

the standard generalization tool for deep networks under Adam/AdamW training, especially LLM pretraining and finetuning; interviews ask for the SGD equivalence derivation, why it breaks under Adam, and AdamW's improvement.

constrains the parameter norm to prevent overfitting. AdamW (Loshchilov & Hutter 2019) fixes the mis-scaled weight decay under Adam: the decay coefficient is constant and independent of gradient magnitude, improving generalization (the paper reports roughly 2% lower test error) and robustness to learning-rate / weight-decay hyperparameters; LLM pretraining (e.g. LLaMA) standardly uses AdamW with weight decay around 0.01–0.1 (an order of magnitude above naive L2's 5e-4).

Derive the equivalence of L2 regularization and weight decay under plain SGD: write both updates $\theta \leftarrow \theta - \eta(\nabla L+\lambda\theta)$ and $\theta \leftarrow (1-\eta\lambda)\theta - \eta\nabla L$ and state the equivalence condition.

Why are L2 and weight decay inequivalent under Adam? What happens when $\lambda\theta$ is divided by $\sqrt{\hat{v}}+\epsilon$?

Write the decoupled AdamW update $\theta \leftarrow \theta - \eta\lambda\theta - \eta\frac{\hat{m}}{\sqrt{\hat{v}}+\epsilon}$ and explain why it generalizes better than Adam + L2.

Hyperparameter scale: what is typical for AdamW weight decay (0.01–0.1)? How does it relate to L2's common 5e-4?

Does weight decay drive weights all the way to zero? Why does it mostly constrain the norm rather than zero out parameters?