M3-041M3: Deep Learning FoundationsLearning Rate SchedulesEasy
⚡Flashcard Drill➔Mastery:

Comparing Learning Rate Schedules: Cosine, Step, Linear, and WSD

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-042 →
📐 Mathematical Definition
cosine: ηt=ηmin⁡+12(η0−ηmin⁡)(1+cos⁡πtT)\text{cosine}:\ \eta_t=\eta_{\min}+\tfrac12(\eta_0-\eta_{\min})\left(1+\cos\tfrac{\pi t}{T}\right)
⚡ One-Sentence Core Answer
Step drops abruptly; Cosine decays smoothly to near-zero; Linear is simple and predictable; WSD maintains a long stable exploration plateau before rapid annealing.

📌 Key Takeaways

  • •
    Step Decay: drops η\eta by factor γ\gamma (e.g., 0.1) at fixed epochs; effective in CNNs, causes sharp validation jumps
  • •
    Cosine Annealing: smooth transition following half-period cosine; standard in modern vision and fixed-budget pre-training
  • •
    WSD (Warmup-Stable-Decay): stable exploration plateau allows arbitrary training extensions before a short 10% decay phase

📐 Mathematical Derivations

Mathematical Formulations:
① Cosine Annealing (Loshchilov & Hutter, 2016):
ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡(t−TwarmTtotal−Twarmπ))\eta_t = \eta_{\min} + \frac{1}{2} (\eta_{\max} - \eta_{\min}) \left( 1 + \cos\left(\frac{t - T_{\text{warm}}}{T_{\text{total}} - T_{\text{warm}}} \pi\right) \right).
Smoothly reduces learning rate to ηmin⁡≈0\eta_{\min} \approx 0, allowing parameters to settle gently into local minima.
② Linear Decay:
ηt=ηmax⁡−(ηmax⁡−ηmin⁡)t−TwarmTtotal−Twarm\eta_t = \eta_{\max} - (\eta_{\max} - \eta_{\min}) \frac{t - T_{\text{warm}}}{T_{\text{total}} - T_{\text{warm}}}. Standard in BERT and early NLP fine-tuning.
③ WSD (Warmup-Stable-Decay, MiniCPM / Hu et al., 2024):
- Warmup (0≤t<Tw0 \le t < T_w): Linear increase to ηmax⁡\eta_{\max}.
- Stable (Tw≤t<TsT_w \le t < T_s): Constant ηmax⁡\eta_{\max} exploring loss basins.
- Decay (Ts≤t≤TT_s \le t \le T): Rapid decay (cosine or 1-sqrt) over final 10–20% of steps to condense representations.
Key Advantage of WSD: Cosine requires pre-committing to total step count TtotalT_{\text{total}}. If you stop early, η\eta did not decay; if you extend, you cannot cleanly resume. WSD allows continuing training indefinitely during the stable phase.

🏭 Production Trade-offs

Schedule selection: For fixed-budget competitive benchmarks, Cosine is standard. For multi-stage foundational LLM pretraining where compute budget may scale dynamically, WSD is rapidly becoming the industry gold standard.
⚠️ Common Interview Pitfalls
  • ✕
    Changing the total step count midway through Cosine Annealing without recomputing the schedule, causing sudden discontinuous learning rate jumps
  • ✕
    Using Step Decay without verifying whether loss plateaus align with the manual step boundaries
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is Cosine Annealing unsuitable for continuous or open-ended pretraining runs?
  • ?
    What empirical evidence demonstrates that 90% of WSD compute can be spent in the stable phase?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-040: Why Learning Rate Warmup is Necessary in Deep Learning📋 Back to BankNext →M3-042: Relationship Between Learning Rate and Batch Size: Linear vs Square-Root Scaling