M3-030M3: Deep Learning FoundationsOptimizers & Second-Order MethodsEasy
⚡Flashcard Drill➔Mastery:

Update Formulas for SGD, SGD with Momentum, and Adam

📐 Mathematical Definition
SGD: θ←θ−ηg;Mom: v←βv+g, θ←θ−ηv;Adam: θ←θ−ηm^v^+ϵ\text{SGD}:\ \theta\leftarrow\theta-\eta g;\qquad \text{Mom}:\ v\leftarrow\beta v+g,\ \theta\leftarrow\theta-\eta v;\qquad \text{Adam}:\ \theta\leftarrow\theta-\eta\frac{\hat m}{\sqrt{\hat v}+\epsilon}
⚡ One-Sentence Core Answer
SGD updates along negative gradient; Momentum adds velocity to escape saddle points; Adam normalizes updates using exponentially decaying first and second gradient moments.

📌 Key Takeaways

  • •
    SGD: θt+1=θt−ηgt\theta_{t+1} = \theta_t - \eta g_t
  • •
    Momentum: vt=βvt−1+gtv_t = \beta v_{t-1} + g_t, θt+1=θt−ηvt\theta_{t+1} = \theta_t - \eta v_t
  • •
    Adam: tracks first moment mtm_t and second moment vtv_t, applying bias correction and adaptive scaling

📐 Mathematical Derivations

Mathematical Formulations:
① Standard SGD:
θt+1=θt−ηgt\theta_{t+1} = \theta_t - \eta g_t, where gt=∇θL(θt)g_t = \nabla_\theta \mathcal{L}(\theta_t).
② SGD with Classical Momentum (Polyak, 1964):
vt=βvt−1+gtv_t = \beta v_{t-1} + g_t, θt+1=θt−ηvt\quad \theta_{t+1} = \theta_t - \eta v_t (or alternatively vt=βvt−1+(1−β)gtv_t = \beta v_{t-1} + (1 - \beta) g_t).
Acts as a physical heavy ball rolling down a potential well, damping high-frequency oscillations across ravines.
③ Adam (Kingma & Ba, 2014):
- First moment: mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t
- Second moment: vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2
- Bias correction: m^t=mt1−β1t\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, v^t=vt1−β2t\quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}
- Parameter update: θt+1=θt−ηv^t+ϵm^t\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t.
Default hyperparameters: β1=0.9,β2=0.999,ϵ=10−8\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8}.

🏭 Production Trade-offs

Optimizer characteristics: SGD+Momentum requires delicate learning rate scheduling and tuning but often achieves lower generalization error in vision tasks. Adam is robust to sparse gradients, scale-invariant across parameters, and converges rapidly across diverse architectures.
⚠️ Common Interview Pitfalls
  • ✕
    Using SGD without momentum on deep networks, resulting in stagnation in ill-conditioned ravines and saddle points
  • ✕
    Assuming Adam replaces learning rate schedules; adaptive optimizers still require cosine/linear decay schedules for optimal convergence
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Momentum accelerate convergence in ravines with high condition numbers?
  • ?
    What role does the second moment vtv_t play in scaling updates for sparse vs frequent parameters?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

Explore the comprehensive technical article, exam cards, and global architecture tree.

← PreviousM3-029: Normalization Operator Fusion in Inference and Its Impact on Serving Latency📋Back to BankNext →M3-031: Adam vs AdamW: Structural Differences and Why AdamW is Mathematically Correct
中文速览与面试要点

写出 SGD、Momentum、Adam 的更新式

SGD 只用一阶梯度;Momentum 累积历史梯度形成速度;Adam 同时用一阶矩(动量)与二阶矩(自适应步长)。

核心要点

  • 数学机理:把三种优化器统一为 θ←θ−η·P⁻¹g 的形式。 SGD :P=I,步长对所有参数相同,故对病态条件数(不同方向曲率差异大)极其敏感——在陡峭方向震荡、在平坦方向爬行。 Momentum :v_t=Σ_kβ^t−kg_k 是历史梯度的指数加权和(EWMA),等价于引入'速度';在方向一致的方向上梯度累加(加速)、在震荡方向上正负抵消(抑制震荡),有效把条件数从 κ 降到约 √κ。 Adam :m_t 为一阶矩估计(动量),v_t 为二阶矩估计(梯度平方的 EWMA);更新量 m̂/√v̂ 中分子是'平均梯度方向'、分母是'该方向的典型梯度幅度',故 更新量级被归一化到约 ±1 (乘以 η),实现逐参数自适应步长——梯度大的参数自动缩小步长、梯度小的放大步长。
  • 深度剖析与工程权衡:① 三者关系 ——Adam 可视为'带自适应步长的 Momentum';去掉 v 的平方根归一化即退化为 Momentum,去掉 m 的动量即退化为 RMSProp。理解这条谱系可快速回答'某优化器是什么'。② 显存账本 ——Adam 需存 m、v 两份状态(FP32 下每参数 8 字节);对 7B 模型即额外约 56 GB,这正是 ZeRO-2/3 分片优化器状态与 8-bit Adam 的动机。③ 条件数与预条件 ——Adam 的对角预条件(每参数独立缩放)只能处理对角曲率差异;真正的最优预条件需要完整 Hessian(见自然梯度题),这是 Adam 与二阶法的本质差距。④ 超参敏感性 ——SGD 对 lr 极度敏感(需要精细调),Adam 相对鲁棒(lr 可在 1e-4~1e-3 间粗调),这是大模型普遍用 Adam 的工程原因之一。⑤ 收敛理论 ——Adam 在凸问题上有 O(1/√T) 的遗憾界(与 SGD 同阶),但 非凸下无收敛保证 (早期 Adam 因 v 的偏差导致不收敛,Reddi 等人指出后才有 AMSGrad 修正);实践中的稳定来自 β₂ 的 EMA 平滑与 ε 的下界保护。⑥ 面试要点 ——被问'Adam 为什么快',答案要落在'逐参数自适应步长 + 动量平滑',而非笼统的'因为它聪明'。
  • Adam 维护 m(一阶矩)与 v(二阶矩)两组状态,显存是参数量的 2 倍
  • Momentum 用 β≈0.9;Adam 用 β₁=0.9、β₂=0.999
  • 三者都可用统一框架 '预条件梯度下降' 理解

常见陷阱

  • 以为 Adam 的 v 是方差(未减均值,实为平方的 EWMA)
  • 忽略 Adam 需要 2 倍参数量的状态显存