Adam/AdamW Math

Adam = exponential moving averages of the first moment $m_t$ and second moment $v_t$, plus bias correction. With gradient $g_t$: $m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$, $v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$. Expanding $m_t$ as a geometric series: $m_t = (1-\beta_1)\sum_{i=1}^{t}\beta_1^{t-i}g_i$, so older gradients get weight $\beta_1^{t-i}$, decaying exponentially. Bias derivation: assuming stationary gradients $\mathbb{E}[g_i] = \mathbb{E}[g]$, we get $\mathbb{E}[m_t] = (1-\beta_1)\mathbb{E}[g]\sum_{k=0}^{t-1}\beta_1^k = (1-\beta_1)\mathbb{E}[g]\cdot\frac{1-\beta_1^t}{1-\beta_1} = (1-\beta_1^t)\mathbb{E}[g]$. For small $t$ this factor is far below 1 (at $t=1$ it is $1-\beta_1 = 0.1$), so early moments are systematically underestimated; dividing by $1-\beta_1^t$ gives $\hat m_t = \frac{m_t}{1-\beta_1^t}$ and $\hat v_t = \frac{v_t}{1-\beta_2^t}$ (since $\mathbb{E}[v_t] = (1-\beta_2^t)\mathbb{E}[g^2]$), so $\mathbb{E}[\hat m_t] = \mathbb{E}[g]$. Full update: $w_{t+1} = w_t - \eta\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}$.

the default optimizer for Transformers, diffusion models and other large-batch / sparse-gradient workloads; interviews frequently ask why bias correction is needed and how AdamW differs from Adam with L2.

vs SGD+momentum, Adam adaptively scales the per-parameter step by $\hat m_t/\sqrt{\hat v_t}$ (scale ~1), which is robust to ill-conditioning ($\kappa$ large) and sparse gradients, while bias correction prevents slow early steps from underestimated moments. AdamW decouples weight decay from L2: in Adam+L2 the term $\lambda w_t$ enters $g_t$ and gets rescaled by $\sqrt{\hat v_t}$, so the effective decay $\eta\lambda w_t/\sqrt{\hat v_t}$ varies with historical gradient magnitude — large-gradient directions are regularized too much, small ones too little; AdamW applies $w_{t+1} = w_t - \eta\left(\lambda w_t + \frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}\right)$, giving every parameter a constant per-step decay factor $1 - \eta\lambda$, which trains more stably and generalizes better.

Test Your Knowledge: Practice Questions for "Adam/AdamW Math"

Single choice pitfall questions with instant feedback and mistake tracking.