M3-046M3: Deep Learning FoundationsLearning Rate SchedulesHard
⚡Flashcard Drill➔Mastery:

The One-Cycle Policy and Cyclical Learning Rates: Mechanisms and Super-Convergence

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-047 →
📐 Mathematical Definition
ηt=ηmin⁡+12(ηmax⁡−ηmin⁡)(1−cos⁡πtT) (one-cycle)\eta_t=\eta_{\min}+\tfrac12(\eta_{\max}-\eta_{\min})\left(1-\cos\frac{\pi t}{T}\right)\ (\text{one-cycle})
⚡ One-Sentence Core Answer
One-Cycle ramps learning rate up to a high peak while dropping momentum, then anneals it to near zero, achieving super-convergence in drastically fewer epochs.

📌 Key Takeaways

  • •
    Super-convergence: Leslie Smith showed models can train up to 5−10×5-10\times faster using aggressive peak learning rates
  • •
    Inverse momentum coupling: as learning rate increases to ηmax⁡\eta_{\max}, momentum decreases from 0.950.95 to 0.850.85, stabilizing aggressive steps
  • •
    Two-phase structure: Phase 1 accelerates exploration and escapes saddle points; Phase 2 anneals learning rate to settle into flat minima

📐 Mathematical Derivations

Mathematical Trajectory (Smith & Topin, 2017):
Let total training steps be TT. Partition into three intervals (typically 45% - 45% - 10%):
1. Phase 1 (Ascent, $0 \to 0.45 T$): Learning rate increases linearly from ηinit=ηmax⁡/10\eta_{\text{init}} = \eta_{\max} / 10 to ηmax⁡\eta_{\max}. Concurrently, momentum decreases linearly from βmax⁡=0.95\beta_{\max} = 0.95 to βmin⁡=0.85\beta_{\min} = 0.85.
2. Phase 2 (Descent, $0.45 T \to 0.90 T$): Learning rate decreases linearly from ηmax⁡\eta_{\max} to ηinit\eta_{\text{init}}. Momentum increases back from 0.850.85 to 0.950.95.
3. Phase 3 (Annihilation, $0.90 T \to T$): Learning rate drops from ηinit\eta_{\text{init}} down to ηmin=ηinit/100\eta_{\text{min}} = \eta_{\text{init}} / 100, allowing fine-grained convergence.
Why it works: High ηmax⁡\eta_{\max} acts as a massive regularizer, preventing the network from overfitting early and enabling it to cross barrier ridges between basins. Lowering momentum when η\eta is high prevents explosive velocity accumulation.

🏭 Production Trade-offs

Application scope: Highly effective for computer vision and fine-tuning classification networks on small-to-medium datasets where rapid training is desired. Rarely used in multi-month foundational LLM pre-training runs.
⚠️ Common Interview Pitfalls
  • ✕
    Setting ηmax⁡\eta_{\max} arbitrarily without running an LR Range Test, causing early divergence during Phase 1
  • ✕
    Using One-Cycle in streaming or continuous learning setups where total step count TT is undefined
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must momentum be decreased when learning rate is increased in the One-Cycle policy?
  • ?
    What is the mathematical mechanism behind 'super-convergence' in deep residual networks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-045: Why Warmup-Stable-Decay (WSD) is Ideal for Large Language Model Pre-training📋 Back to BankNext →M3-047: Major Regularization Techniques in Deep Learning