Learning Rate Scheduling

LR scheduling changes the learning rate $\eta_t$ over training according to a plan. Four mainstream strategies: ① Warmup (linear): $\eta_t = \eta_{\max}\cdot\frac{t}{T_{\text{warm}}}$ for $t = 1 \ldots T_{\text{warm}}$, ramping linearly from near zero to the peak $\eta_{\max}$; ② linear decay: $\eta_t = \eta_{\max}\left(1 - \frac{t}{T}\right)$; ③ step decay: $\eta_t = \eta_0\gamma^{\lfloor t/s \rfloor}$, multiplying by $\gamma$ (e.g. ×0.1) every $s$ steps; ④ cosine annealing: $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$ — at $t=0$, $\eta_0 = \eta_{\max}$; at $t=T$, $\eta_T = \eta_{\min}$, a smooth drop with no cliff. Large-model training standardizes on “warmup + cosine”: 1%–5% of steps ramp linearly to the peak, then cosine decays smoothly to a tiny final value. Why warmup: early in training Adam's $v_t$ has seen only a few gradients, so $\hat{v}_t$ is tiny and $\eta/\sqrt{\hat{v}_t}$ can be enormous — one step can blow the parameters away (worse with large batches and noisy gradients); a small LR lets the moment estimates and BN/LN statistics stabilize first.

a standard component of every SOTA large-model and CNN training pipeline; interview favorites: why warmup is necessary, cosine vs step decay, and the LR–batch-size scaling rule.

a fixed LR faces a dilemma — large LR converges fast early but oscillates or diverges near the optimum, small LR is too slow early; scheduling hard-codes “large steps to explore, small steps to refine”. Cosine annealing approaches a tiny LR smoothly, avoiding the loss spikes of step-decay cliffs, and often lands in flatter minima with better generalization. Linear scaling rule (Goyal 2017): doubling the batch size roughly doubles the LR (larger batches have less gradient noise, allowing larger effective steps), but only works with warmup, otherwise early steps diverge.

Write the linear warmup $\eta_t = \eta_{\max}\cdot\frac{t}{T_{\text{warm}}}$ and cosine annealing $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max}-\eta_{\min})\left(1+\cos\left(\frac{t}{T}\pi\right)\right)$; what is $\eta_t$ at $t=0$ and $t=T$?

Why is warmup needed at the start? Derive why $\eta/\sqrt{\hat{v}_t}$ can be huge early in Adam (only a few gradients have accumulated into $\hat{v}_t$) and why large batches make it riskier.

Cosine annealing vs step decay: why is a smooth decline better (avoiding loss spikes from cliffs)? What does a tiny final LR do (converging to flat minima, better generalization)?

Linear scaling rule (Goyal): why does doubling the batch size roughly double the LR? How does gradient noise scale with batch size? Why must warmup accompany it?

Typical large-model config: warmup fraction (1%–5%), peak LR, cosine endpoint; how do the schedule and the optimizer (Adam's $\beta_2$, $\epsilon$) interact?