Adam & AdamW

Adam (adaptive moment estimation) maintains exponential moving averages of the first moment $m_t$ and the second moment (uncentered variance) $v_t$ of the gradient, with bias correction: $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$, $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$; bias correction $\hat{m}_t = \frac{m_t}{1-\beta_1^t}$, $\hat{v}_t = \frac{v_t}{1-\beta_2^t}$ (the geometric series gives $\mathbb{E}[m_t] = (1-\beta_1^t)\mathbb{E}[g]$; for small $t$ the moments are systematically underestimated, and dividing by $1-\beta_1^t$ fixes this); parameter update $\theta_{t+1} = \theta_t - \eta\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$. Defaults: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, base LR $\eta = 10^{-3}$. The ratio $\hat{m}_t/\sqrt{\hat{v}_t}$ has magnitude ~1, i.e. per-parameter normalization by gradient RMS: large-gradient directions are scaled down, small ones up, making Adam robust to sparse gradients and ill-conditioning (large $\kappa$). $\epsilon$ belongs in the denominator: numerical stability (avoid division by zero when $\hat{v}_t \approx 0$ early in training) and an implicit step bound $\eta/\sqrt{\epsilon}$. AdamW decouples weight decay from L2: $\theta_{t+1} = \theta_t - \eta\left(\lambda\theta_t + \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}\right)$, giving every parameter a constant per-step decay factor $1 - \eta\lambda$.

the default optimizer for Transformers, BERT/GPT and other large models; the first choice for large batches, sparse features and multimodal workloads; interviews always ask to write the four formulas, explain bias correction, quote the default $\beta$s, and contrast AdamW with Adam+L2.

SGD+momentum shares one learning rate across all parameters, which cannot fit ill-conditioned or sparse gradients; Adam adaptively scales the per-parameter step and bias correction prevents early iterations from being slowed by underestimated moments. AdamW fixes Adam+L2: the L2 term $\lambda w_t$ enters $g_t$ and is rescaled by $\sqrt{\hat{v}_t}$, so the effective decay $\eta\lambda w_t/\sqrt{\hat{v}_t}$ varies with gradient magnitude — large-gradient directions are regularized too much, small ones too little; decoupling yields a constant decay rate per parameter, training more stably and generalizing better.

Write Adam's four formulas (moment recursions, bias correction, parameter update) and quote the defaults $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, $\eta = 10^{-3}$.

Why is bias correction needed? Sum the geometric series to prove $\mathbb{E}[m_t] = (1-\beta_1^t)\mathbb{E}[g]$, explain why the bias is worst at $t=1$ ($1-\beta_1 = 0.1$), and why dividing by $1-\beta_1^t$ gives $\mathbb{E}[\hat{m}_t] = \mathbb{E}[g]$.

Why update by $\hat{m}_t/\sqrt{\hat{v}_t}$? Relation to RMSProp/AdaGrad; why must $\epsilon$ sit in the denominator (numerical stability + step bound $\eta/\sqrt{\epsilon}$)?

Adam vs SGD+momentum: use cases and generalization differences (why SGD often wins on CV while large models standardize on Adam); why Adam converges fast early but can be unstable later.

Write the AdamW update and compare with Adam+L2: how does dividing the L2 term by $\sqrt{\hat{v}_t}$ distort the effective regularization? Why does decoupling give a constant per-step decay $1 - \eta\lambda$?