Learning Rate Scheduling

LR scheduling changes the learning rate $\eta_t$ over training according to a plan. Four mainstream strategies: ① Warmup (linear): $\eta_t = \eta_{\max}\cdot\frac{t}{T_{\text{warm}}}$ for $t = 1 \ldots T_{\text{warm}}$, ramping linearly from near zero to the peak $\eta_{\max}$; ② linear decay: $\eta_t = \eta_{\max}\left(1 - \frac{t}{T}\right)$; ③ step decay: $\eta_t = \eta_0\gamma^{\lfloor t/s \rfloor}$, multiplying by $\gamma$ (e.g. ×0.1) every $s$ steps; ④ cosine annealing: $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})\left(1 + \cos\left(\frac{t}{T}\pi\right)\right)$ — at $t=0$, $\eta_0 = \eta_{\max}$; at $t=T$, $\eta_T = \eta_{\min}$, a smooth drop with no cliff. Large-model training standardizes on “warmup + cosine”: 1%–5% of steps ramp linearly to the peak, then cosine decays smoothly to a tiny final value. Why warmup: early in training Adam's $v_t$ has seen only a few gradients, so $\hat{v}_t$ is tiny and $\eta/\sqrt{\hat{v}_t}$ can be enormous — one step can blow the parameters away (worse with large batches and noisy gradients); a small LR lets the moment estimates and BN/LN statistics stabilize first.

a standard component of every SOTA large-model and CNN training pipeline; interview favorites: why warmup is necessary, cosine vs step decay, and the LR–batch-size scaling rule.

a fixed LR faces a dilemma — large LR converges fast early but oscillates or diverges near the optimum, small LR is too slow early; scheduling hard-codes “large steps to explore, small steps to refine”. Cosine annealing approaches a tiny LR smoothly, avoiding the loss spikes of step-decay cliffs, and often lands in flatter minima with better generalization. Linear scaling rule (Goyal 2017): doubling the batch size roughly doubles the LR (larger batches have less gradient noise, allowing larger effective steps), but only works with warmup, otherwise early steps diverge.

Test Your Knowledge: Practice Questions for "Learning Rate Scheduling"

Single choice pitfall questions with instant feedback and mistake tracking.