M3-045M3: Deep Learning FoundationsLearning Rate SchedulesMedium
⚡Flashcard Drill➔Mastery:
Why Warmup-Stable-Decay (WSD) is Ideal for Large Language Model Pre-training
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
WSD maintains a long stable exploration plateau at peak learning rate, allowing checkpoints to be continuously resumed or extended before a brief final annealing phase.
📌 Key Takeaways
- •Decoupled budget: unlike Cosine, WSD does not require knowing the total token/step budget in advance
- •Loss basin exploration: high stable learning rate explores diverse parameter valleys continuously
- •Annealing flexibility: whenever compute budget ends, branch a short decay run ( of steps) to achieve peak convergence
📐 Mathematical Derivations
Mechanisms and Scaling Laws (Hu et al., MiniCPM / LLaMA-style continuous pre-training):
Standard Cosine Annealing couples the decay curve strictly to total budget : . If an organization decides to train a model for 2 trillion tokens instead of 1 trillion, a model trained with Cosine must either be restarted or accept a suboptimal broken decay profile.
WSD 3-Phase Schedule:
1. Warmup ($0 \to T_w$): Linear ramp to peak learning rate ( of steps).
2. Stable Phase ($T_w \to T_s$): Maintain constant high learning rate for the majority () of the run.
- In this phase, loss plateaus at a moderate level while parameters explore the wide flat basin.
- Training can continue indefinitely across dataset expansions or hardware additions.
3. Decay Phase ($T_s \to T_{\text{final}}$): Rapid annealing (e.g., exponential or 1-sqrt decay) over the final of steps down to or 0.
- Empirical finding: A 10% decay phase achieves identical or superior downstream benchmark performance compared to a full-run cosine schedule.
Standard Cosine Annealing couples the decay curve strictly to total budget : . If an organization decides to train a model for 2 trillion tokens instead of 1 trillion, a model trained with Cosine must either be restarted or accept a suboptimal broken decay profile.
WSD 3-Phase Schedule:
1. Warmup ($0 \to T_w$): Linear ramp to peak learning rate ( of steps).
2. Stable Phase ($T_w \to T_s$): Maintain constant high learning rate for the majority () of the run.
- In this phase, loss plateaus at a moderate level while parameters explore the wide flat basin.
- Training can continue indefinitely across dataset expansions or hardware additions.
3. Decay Phase ($T_s \to T_{\text{final}}$): Rapid annealing (e.g., exponential or 1-sqrt decay) over the final of steps down to or 0.
- Empirical finding: A 10% decay phase achieves identical or superior downstream benchmark performance compared to a full-run cosine schedule.
🏭 Production Trade-offs
Industrial value: WSD enables multi-stage data curation. Teams can inject higher-quality code or synthetic reasoning data specifically during the final decay phase, achieving massive reasoning jumps without retraining from scratch.
⚠️ Common Interview Pitfalls
- ✕Making the decay phase too short ( of steps), preventing parameters from condensing into sharp minima
- ✕Evaluating model capabilities during the stable phase, falsely concluding that the model has plateaued before annealing
🎯 Interviewer Follow-up Inquiries
- ?Why does injecting high-quality synthetic data during the WSD decay phase yield disproportionate performance gains?
- ?How does WSD compare with Cosine Annealing when the total compute budget is strictly fixed in advance?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚