M3-046M3: Deep Learning FoundationsLearning Rate SchedulesHard
⚡Flashcard Drill➔Mastery:
The One-Cycle Policy and Cyclical Learning Rates: Mechanisms and Super-Convergence
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
One-Cycle ramps learning rate up to a high peak while dropping momentum, then anneals it to near zero, achieving super-convergence in drastically fewer epochs.
📌 Key Takeaways
- •Super-convergence: Leslie Smith showed models can train up to faster using aggressive peak learning rates
- •Inverse momentum coupling: as learning rate increases to , momentum decreases from to , stabilizing aggressive steps
- •Two-phase structure: Phase 1 accelerates exploration and escapes saddle points; Phase 2 anneals learning rate to settle into flat minima
📐 Mathematical Derivations
Mathematical Trajectory (Smith & Topin, 2017):
Let total training steps be . Partition into three intervals (typically 45% - 45% - 10%):
1. Phase 1 (Ascent, $0 \to 0.45 T$): Learning rate increases linearly from to . Concurrently, momentum decreases linearly from to .
2. Phase 2 (Descent, $0.45 T \to 0.90 T$): Learning rate decreases linearly from to . Momentum increases back from to .
3. Phase 3 (Annihilation, $0.90 T \to T$): Learning rate drops from down to , allowing fine-grained convergence.
Why it works: High acts as a massive regularizer, preventing the network from overfitting early and enabling it to cross barrier ridges between basins. Lowering momentum when is high prevents explosive velocity accumulation.
Let total training steps be . Partition into three intervals (typically 45% - 45% - 10%):
1. Phase 1 (Ascent, $0 \to 0.45 T$): Learning rate increases linearly from to . Concurrently, momentum decreases linearly from to .
2. Phase 2 (Descent, $0.45 T \to 0.90 T$): Learning rate decreases linearly from to . Momentum increases back from to .
3. Phase 3 (Annihilation, $0.90 T \to T$): Learning rate drops from down to , allowing fine-grained convergence.
Why it works: High acts as a massive regularizer, preventing the network from overfitting early and enabling it to cross barrier ridges between basins. Lowering momentum when is high prevents explosive velocity accumulation.
🏭 Production Trade-offs
Application scope: Highly effective for computer vision and fine-tuning classification networks on small-to-medium datasets where rapid training is desired. Rarely used in multi-month foundational LLM pre-training runs.
⚠️ Common Interview Pitfalls
- ✕Setting arbitrarily without running an LR Range Test, causing early divergence during Phase 1
- ✕Using One-Cycle in streaming or continuous learning setups where total step count is undefined
🎯 Interviewer Follow-up Inquiries
- ?Why must momentum be decreased when learning rate is increased in the One-Cycle policy?
- ?What is the mathematical mechanism behind 'super-convergence' in deep residual networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚