Adam & AdamW

Adam (adaptive moment estimation) maintains exponential moving averages of the first moment $m_t$ and the second moment (uncentered variance) $v_t$ of the gradient, with bias correction: $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$, $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$; bias correction $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$, $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$ (the geometric series gives $\mathbb{E}[m_t] = (1-\beta_1^t)\mathbb{E}[g]$; for small $t$ the moments are systematically underestimated, and dividing by $1-\beta_1^t$ fixes this); parameter update $\theta_{t+1} = \theta_t - \eta\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$. Defaults: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, base LR $\eta = 10^{-3}$. The ratio $\hat{m}_t/\sqrt{\hat{v}_t}$ has magnitude ~1, i.e. per-parameter normalization by gradient RMS: large-gradient directions are scaled down, small ones up, making Adam robust to sparse gradients and ill-conditioning (large $\kappa$). $\epsilon$ belongs in the denominator: numerical stability (avoid division by zero when $\hat{v}_t \approx 0$ early in training) and an implicit step bound $\eta/\sqrt{\epsilon}$. AdamW decouples weight decay from L2: $\theta_{t+1} = \theta_t - \eta\left(\lambda\theta_t + \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}\right)$, giving every parameter a constant per-step decay factor $1 - \eta\lambda$.

the default optimizer for Transformers, BERT/GPT and other large models; the first choice for large batches, sparse features and multimodal workloads; interviews always ask to write the four formulas, explain bias correction, quote the default $\beta$s, and contrast AdamW with Adam+L2.

SGD+momentum shares one learning rate across all parameters, which cannot fit ill-conditioned or sparse gradients; Adam adaptively scales the per-parameter step and bias correction prevents early iterations from being slowed by underestimated moments. AdamW fixes Adam+L2: the L2 term $\lambda w_t$ enters $g_t$ and is rescaled by $\sqrt{\hat{v}_t}$, so the effective decay $\eta\lambda w_t/\sqrt{\hat{v}_t}$ varies with gradient magnitude — large-gradient directions are regularized too much, small ones too little; decoupling yields a constant decay rate per parameter, training more stably and generalizing better.

Test Your Knowledge: Practice Questions for "Adam & AdamW"

Single choice pitfall questions with instant feedback and mistake tracking.