M3-039M3: Deep Learning FoundationsOptimizers & Second-Order MethodsMedium
⚡Flashcard Drill➔Mastery:
Comparing Adagrad, RMSProp, and AdaDelta: The Diminishing Learning Rate Problem
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Adagrad accumulates all historical squared gradients causing premature learning rate collapse; RMSProp replaces sum with exponential moving average to preserve adaptive learning.
📌 Key Takeaways
- •Adagrad: ; monotonically increasing drives effective learning rate to zero prematurely
- •RMSProp: ; leaky moving average discards distant history, enabling continuous adaptation
- •AdaDelta: eliminates learning rate hyperparameter entirely by tracking parameter displacement root-mean-square
📐 Mathematical Derivations
Evolution of Adaptive Optimizers:
① Adagrad (Duchi et al., 2011):
. Update: .
- Advantage: Automatically assigns smaller steps to frequent features and larger steps to rare features (ideal for sparse text/recommendations).
- Defect: Because , is monotonically non-decreasing. As , , causing the effective step size . Training stalls prematurely.
② RMSProp (Hinton, 2012):
Replaces infinite sum with an Exponential Moving Average (EMA): .
Update: . Leaky decay over effective window bounds , sustaining training indefinitely.
③ AdaDelta (Zeiler, 2012):
Fixes unit mismatch by scaling updates by running RMS of parameter updates: , eliminating learning rate completely.
① Adagrad (Duchi et al., 2011):
. Update: .
- Advantage: Automatically assigns smaller steps to frequent features and larger steps to rare features (ideal for sparse text/recommendations).
- Defect: Because , is monotonically non-decreasing. As , , causing the effective step size . Training stalls prematurely.
② RMSProp (Hinton, 2012):
Replaces infinite sum with an Exponential Moving Average (EMA): .
Update: . Leaky decay over effective window bounds , sustaining training indefinitely.
③ AdaDelta (Zeiler, 2012):
Fixes unit mismatch by scaling updates by running RMS of parameter updates: , eliminating learning rate completely.
🏭 Production Trade-offs
Modern usage: Adagrad remains competitive for massive sparse embeddings in search and recommendation models (DLRM). RMSProp is standard in reinforcement learning (DQN, A3C). Adam combines RMSProp's second moment with classical momentum.
⚠️ Common Interview Pitfalls
- ✕Using Adagrad for training deep networks across many epochs, where learning rate collapse halts progress early
- ✕Assuming AdaDelta requires zero tuning; while is absent, decay momentum and still influence dynamics
🎯 Interviewer Follow-up Inquiries
- ?Why is Adagrad still widely used for large sparse embedding tables in recommendation systems?
- ?How does AdaDelta correct the physical unit mismatch present in standard gradient descent?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚