M3-030M3: Deep Learning FoundationsOptimizers & Second-Order MethodsEasy
⚡Flashcard Drill➔Mastery:

Update Formulas for SGD, SGD with Momentum, and Adam

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-031 →
📐 Mathematical Definition
SGD: θ←θ−ηg;Mom: v←βv+g, θ←θ−ηv;Adam: θ←θ−ηm^v^+ϵ\text{SGD}:\ \theta\leftarrow\theta-\eta g;\qquad \text{Mom}:\ v\leftarrow\beta v+g,\ \theta\leftarrow\theta-\eta v;\qquad \text{Adam}:\ \theta\leftarrow\theta-\eta\frac{\hat m}{\sqrt{\hat v}+\epsilon}
⚡ One-Sentence Core Answer
SGD updates along negative gradient; Momentum adds velocity to escape saddle points; Adam normalizes updates using exponentially decaying first and second gradient moments.

📌 Key Takeaways

  • •
    SGD: θt+1=θt−ηgt\theta_{t+1} = \theta_t - \eta g_t
  • •
    Momentum: vt=βvt−1+gtv_t = \beta v_{t-1} + g_t, θt+1=θt−ηvt\theta_{t+1} = \theta_t - \eta v_t
  • •
    Adam: tracks first moment mtm_t and second moment vtv_t, applying bias correction and adaptive scaling

📐 Mathematical Derivations

Mathematical Formulations:
① Standard SGD:
θt+1=θt−ηgt\theta_{t+1} = \theta_t - \eta g_t, where gt=∇θL(θt)g_t = \nabla_\theta \mathcal{L}(\theta_t).
② SGD with Classical Momentum (Polyak, 1964):
vt=βvt−1+gtv_t = \beta v_{t-1} + g_t, θt+1=θt−ηvt\quad \theta_{t+1} = \theta_t - \eta v_t (or alternatively vt=βvt−1+(1−β)gtv_t = \beta v_{t-1} + (1 - \beta) g_t).
Acts as a physical heavy ball rolling down a potential well, damping high-frequency oscillations across ravines.
③ Adam (Kingma & Ba, 2014):
- First moment: mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t
- Second moment: vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
- Bias correction: m^t=mt1−β1t\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, v^t=vt1−β2t\quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}
- Parameter update: θt+1=θt−ηv^t+ϵm^t\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t.
Default hyperparameters: β1=0.9,β2=0.999,ϵ=10−8\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8}.

🏭 Production Trade-offs

Optimizer characteristics: SGD+Momentum requires delicate learning rate scheduling and tuning but often achieves lower generalization error in vision tasks. Adam is robust to sparse gradients, scale-invariant across parameters, and converges rapidly across diverse architectures.
⚠️ Common Interview Pitfalls
  • ✕
    Using SGD without momentum on deep networks, resulting in stagnation in ill-conditioned ravines and saddle points
  • ✕
    Assuming Adam replaces learning rate schedules; adaptive optimizers still require cosine/linear decay schedules for optimal convergence
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Momentum accelerate convergence in ravines with high condition numbers?
  • ?
    What role does the second moment vtv_t play in scaling updates for sparse vs frequent parameters?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-029: Normalization Operator Fusion in Inference and Its Impact on Serving Latency📋 Back to BankNext →M3-031: Adam vs AdamW: Structural Differences and Why AdamW is Mathematically Correct