M3-039M3: Deep Learning FoundationsOptimizers & Second-Order MethodsMedium
⚡Flashcard Drill➔Mastery:

Comparing Adagrad, RMSProp, and AdaDelta: The Diminishing Learning Rate Problem

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-040 →
📐 Mathematical Definition
Adagrad: vt=vt−1+gt2;RMSProp: vt=βvt−1+(1−β)gt2;AdaDelta: Δθ=−RMS[Δθ]t−1RMS[g]tgt\text{Adagrad}:\ v_t=v_{t-1}+g_t^2;\qquad \text{RMSProp}:\ v_t=\beta v_{t-1}+(1-\beta)g_t^2;\qquad \text{AdaDelta}:\ \Delta\theta=-\frac{\mathrm{RMS}[\Delta\theta]_{t-1}}{\mathrm{RMS}[g]_t}g_t
⚡ One-Sentence Core Answer
Adagrad accumulates all historical squared gradients causing premature learning rate collapse; RMSProp replaces sum with exponential moving average to preserve adaptive learning.

📌 Key Takeaways

  • •
    Adagrad: vt=vt−1+gt2v_t = v_{t-1} + g_t^2; monotonically increasing vtv_t drives effective learning rate η/vt\eta / \sqrt{v_t} to zero prematurely
  • •
    RMSProp: vt=βvt−1+(1−β)gt2v_t = \beta v_{t-1} + (1 - \beta) g_t^2; leaky moving average discards distant history, enabling continuous adaptation
  • •
    AdaDelta: eliminates learning rate hyperparameter entirely by tracking parameter displacement root-mean-square

📐 Mathematical Derivations

Evolution of Adaptive Optimizers:
① Adagrad (Duchi et al., 2011):
Gt=Gt−1+gt2=∑τ=1tgτ2G_t = G_{t-1} + g_t^2 = \sum_{\tau=1}^t g_\tau^2. Update: θt+1=θt−ηGt+ϵgt\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t} + \epsilon} g_t.
- Advantage: Automatically assigns smaller steps to frequent features and larger steps to rare features (ideal for sparse text/recommendations).
- Defect: Because gt2≥0g_t^2 \ge 0, GtG_t is monotonically non-decreasing. As t→∞t \to \infty, Gt→∞G_t \to \infty, causing the effective step size ηGt→0\frac{\eta}{\sqrt{G_t}} \to 0. Training stalls prematurely.
② RMSProp (Hinton, 2012):
Replaces infinite sum with an Exponential Moving Average (EMA): vt=βvt−1+(1−β)gt2v_t = \beta v_{t-1} + (1 - \beta) g_t^2.
Update: θt+1=θt−ηvt+ϵgt\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t} + \epsilon} g_t. Leaky decay over effective window 1/(1−β)1/(1-\beta) bounds vtv_t, sustaining training indefinitely.
③ AdaDelta (Zeiler, 2012):
Fixes unit mismatch by scaling updates by running RMS of parameter updates: Δθt=−RMS(Δθ)t−1RMS(g)tgt\Delta \theta_t = - \frac{\text{RMS}(\Delta \theta)_{t-1}}{\text{RMS}(g)_t} g_t, eliminating learning rate η\eta completely.

🏭 Production Trade-offs

Modern usage: Adagrad remains competitive for massive sparse embeddings in search and recommendation models (DLRM). RMSProp is standard in reinforcement learning (DQN, A3C). Adam combines RMSProp's second moment with classical momentum.
⚠️ Common Interview Pitfalls
  • ✕
    Using Adagrad for training deep networks across many epochs, where learning rate collapse halts progress early
  • ✕
    Assuming AdaDelta requires zero tuning; while η\eta is absent, decay momentum ρ\rho and ϵ\epsilon still influence dynamics
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is Adagrad still widely used for large sparse embedding tables in recommendation systems?
  • ?
    How does AdaDelta correct the physical unit mismatch present in standard gradient descent?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-038: Memory-Efficient Optimizers: Adafactor and 8-bit Adam Explained📋 Back to BankNext →M3-040: Why Learning Rate Warmup is Necessary in Deep Learning