M3-030M3: Deep Learning FoundationsOptimizers & Second-Order MethodsEasy
⚡Flashcard Drill➔Mastery:
Update Formulas for SGD, SGD with Momentum, and Adam
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
SGD updates along negative gradient; Momentum adds velocity to escape saddle points; Adam normalizes updates using exponentially decaying first and second gradient moments.
📌 Key Takeaways
- •SGD:
- •Momentum: ,
- •Adam: tracks first moment and second moment , applying bias correction and adaptive scaling
📐 Mathematical Derivations
Mathematical Formulations:
① Standard SGD:
, where .
② SGD with Classical Momentum (Polyak, 1964):
, (or alternatively ).
Acts as a physical heavy ball rolling down a potential well, damping high-frequency oscillations across ravines.
③ Adam (Kingma & Ba, 2014):
- First moment:
- Second moment:
- Bias correction: ,
- Parameter update: .
Default hyperparameters: .
① Standard SGD:
, where .
② SGD with Classical Momentum (Polyak, 1964):
, (or alternatively ).
Acts as a physical heavy ball rolling down a potential well, damping high-frequency oscillations across ravines.
③ Adam (Kingma & Ba, 2014):
- First moment:
- Second moment:
- Bias correction: ,
- Parameter update: .
Default hyperparameters: .
🏭 Production Trade-offs
Optimizer characteristics: SGD+Momentum requires delicate learning rate scheduling and tuning but often achieves lower generalization error in vision tasks. Adam is robust to sparse gradients, scale-invariant across parameters, and converges rapidly across diverse architectures.
⚠️ Common Interview Pitfalls
- ✕Using SGD without momentum on deep networks, resulting in stagnation in ill-conditioned ravines and saddle points
- ✕Assuming Adam replaces learning rate schedules; adaptive optimizers still require cosine/linear decay schedules for optimal convergence
🎯 Interviewer Follow-up Inquiries
- ?Why does Momentum accelerate convergence in ravines with high condition numbers?
- ?What role does the second moment play in scaling updates for sparse vs frequent parameters?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚