LR Warmup & Cosine Annealing Tuning

Learning Rate Warmup & Cosine Annealing Dynamic Scheduling governs the temporal trajectory of optimization step sizes in deep neural networks and LLMs to ensure early stability and late-stage convergence into flat minima; it operates in two contiguous phases: 1) Linear Warmup: ramping the learning rate from 0 to $\eta_{\max}$ over the initial $N_{\text{warmup}}$ steps, preventing chaotic updates when Adam's second-moment variance estimates $v_t$ are still uncalibrated; 2) Cosine Annealing Decay: smoothly decaying the learning rate along a half-cosine period $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_{\max} - \eta_{\min})(1 + \cos(\frac{t - N_{\text{warmup}}}{T_{\text{total}} - N_{\text{warmup}}} \pi))$ down to $\eta_{\min}$, guiding weights into wide, robust basins.

Large language model pre-training, fine-tuning Vision Transformers, and resolving early loss spikes or training stagnation.

Starting immediately with high learning rates causes early gradient instability and NaN crashes; step decays cause sharp gradient shocks; Warmup and Cosine schedules yield seamless, optimal convergence.

Derive the Cosine Annealing schedule equation and explain why smooth half-cosine decay favors flat minima over abrupt step decays?

Why is Linear Warmup mandatory with AdamW (explaining uncalibrated second-moment variance $v_t$ in early iterations)?

Explain the theoretical justification and breakdown limits of the Linear Scaling Rule (scaling learning rate proportionally with batch size)?

Explain Leslie Smith's 1Cycle policy and how inverse momentum coupling achieves Super-Convergence in fewer epochs?

How should learning rates be rewarmed and decayed during Continual Pre-training on new domain corpora?