M3-041M3: Deep Learning FoundationsLearning Rate SchedulesEasy
⚡Flashcard Drill➔Mastery:
Comparing Learning Rate Schedules: Cosine, Step, Linear, and WSD
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Step drops abruptly; Cosine decays smoothly to near-zero; Linear is simple and predictable; WSD maintains a long stable exploration plateau before rapid annealing.
📌 Key Takeaways
- •Step Decay: drops by factor (e.g., 0.1) at fixed epochs; effective in CNNs, causes sharp validation jumps
- •Cosine Annealing: smooth transition following half-period cosine; standard in modern vision and fixed-budget pre-training
- •WSD (Warmup-Stable-Decay): stable exploration plateau allows arbitrary training extensions before a short 10% decay phase
📐 Mathematical Derivations
Mathematical Formulations:
① Cosine Annealing (Loshchilov & Hutter, 2016):
.
Smoothly reduces learning rate to , allowing parameters to settle gently into local minima.
② Linear Decay:
. Standard in BERT and early NLP fine-tuning.
③ WSD (Warmup-Stable-Decay, MiniCPM / Hu et al., 2024):
- Warmup (): Linear increase to .
- Stable (): Constant exploring loss basins.
- Decay (): Rapid decay (cosine or 1-sqrt) over final 10–20% of steps to condense representations.
Key Advantage of WSD: Cosine requires pre-committing to total step count . If you stop early, did not decay; if you extend, you cannot cleanly resume. WSD allows continuing training indefinitely during the stable phase.
① Cosine Annealing (Loshchilov & Hutter, 2016):
.
Smoothly reduces learning rate to , allowing parameters to settle gently into local minima.
② Linear Decay:
. Standard in BERT and early NLP fine-tuning.
③ WSD (Warmup-Stable-Decay, MiniCPM / Hu et al., 2024):
- Warmup (): Linear increase to .
- Stable (): Constant exploring loss basins.
- Decay (): Rapid decay (cosine or 1-sqrt) over final 10–20% of steps to condense representations.
Key Advantage of WSD: Cosine requires pre-committing to total step count . If you stop early, did not decay; if you extend, you cannot cleanly resume. WSD allows continuing training indefinitely during the stable phase.
🏭 Production Trade-offs
Schedule selection: For fixed-budget competitive benchmarks, Cosine is standard. For multi-stage foundational LLM pretraining where compute budget may scale dynamically, WSD is rapidly becoming the industry gold standard.
⚠️ Common Interview Pitfalls
- ✕Changing the total step count midway through Cosine Annealing without recomputing the schedule, causing sudden discontinuous learning rate jumps
- ✕Using Step Decay without verifying whether loss plateaus align with the manual step boundaries
🎯 Interviewer Follow-up Inquiries
- ?Why is Cosine Annealing unsuitable for continuous or open-ended pretraining runs?
- ?What empirical evidence demonstrates that 90% of WSD compute can be spent in the stable phase?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚