🧠 Deep LearningID: sgd-momentum

SGD & Momentum

Q: Write the momentum update $v_t = \gamma v_{t-1} + \eta g_t$, $\theta_{t+1} = \theta_t - v_t$; expand $v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}$ and explain how historical gradients decay by $\gamma^k$.

【SGD & Momentum 核心考点 1】Write the momentum update $v_t = \gamma v_{t-1} + \eta g_t$, $\theta_{t+1} = \theta_t - v_t$; expand $v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}$ and explain how historical gradients decay by $\gamma^k$.。技术解析：Definition: Plain SGD updates along the negative instantaneous gradient: \theta_{t+1} = \theta_t - \eta g_t with g_t = \nabla_\theta L(\theta_t); on ill-conditioned (elliptical) contours with large condition number \kappa = \lambda_{\max}/\lambda_{\min}, gradient directions flip ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/sgd-momentum

Q: Why does momentum suppress oscillation and speed up convergence? Explain via the geometric weighted sum: consistent gradients accumulate, high-frequency reversals cancel, and why this helps on contours with large condition number $\kappa$.

【SGD & Momentum 核心考点 2】Why does momentum suppress oscillation and speed up convergence? Explain via the geometric weighted sum: consistent gradients accumulate, high-frequency reversals cancel, and why this helps on contours with large condition number $\kappa$.。技术解析：Definition: Plain SGD updates along the negative instantaneous gradient: \theta_{t+1} = \theta_t - \eta g_t with g_t = \nabla_\theta L(\theta_t); on ill-conditioned (elliptical) contours with large condition number \kappa = \lambda_{\max}/\lambda_{\min}, gradient directions flip ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/sgd-momentum

Q: Write the NAG update $v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})$; how does the look-ahead step differ from vanilla momentum, and why does the convex rate improve from $O(1/t)$ to $O(1/t^2)$?

【SGD & Momentum 核心考点 3】Write the NAG update $v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})$; how does the look-ahead step differ from vanilla momentum, and why does the convex rate improve from $O(1/t)$ to $O(1/t^2)$?。技术解析：Definition: Plain SGD updates along the negative instantaneous gradient: \theta_{t+1} = \theta_t - \eta g_t with g_t = \nabla_\theta L(\theta_t); on ill-conditioned (elliptical) contours with large condition number \kappa = \lambda_{\max}/\lambda_{\min}, gradient directions flip ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/sgd-momentum

Q: What does the default $\gamma = 0.9$ mean? What problems arise from too large / too small $\gamma$ (overshoot vs no memory)? Why is the effective step under constant gradient $\eta/(1-\gamma)$ (~10x amplification)?

【SGD & Momentum 核心考点 4】What does the default $\gamma = 0.9$ mean? What problems arise from too large / too small $\gamma$ (overshoot vs no memory)? Why is the effective step under constant gradient $\eta/(1-\gamma)$ (~10x amplification)?。技术解析：Definition: Plain SGD updates along the negative instantaneous gradient: \theta_{t+1} = \theta_t - \eta g_t with g_t = \nabla_\theta L(\theta_t); on ill-conditioned (elliptical) contours with large condition number \kappa = \lambda_{\max}/\lambda_{\min}, gradient directions flip ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/sgd-momentum

Q: How does momentum interact with LR scheduling? Why does momentum usually require LR decay (amplified effective step, end-of-training oscillation)? What is the single-learning-rate limitation of momentum vs Adam?

【SGD & Momentum 核心考点 5】How does momentum interact with LR scheduling? Why does momentum usually require LR decay (amplified effective step, end-of-training oscillation)? What is the single-learning-rate limitation of momentum vs Adam?。技术解析：Definition: Plain SGD updates along the negative instantaneous gradient: \theta_{t+1} = \theta_t - \eta g_t with g_t = \nabla_\theta L(\theta_t); on ill-conditioned (elliptical) contours with large condition number \kappa = \lambda_{\max}/\lambda_{\min}, gradient directions flip ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/sgd-momentum

SGD 与动量

🎯Core Definition

Plain SGD updates along the negative instantaneous gradient:

\theta_{t+1} = \theta_t - \eta g_t

with

g_t = \nabla_\theta L(\theta_t)

; on ill-conditioned (elliptical) contours with large condition number

\kappa = \lambda_{\max}/\lambda_{\min}

, gradient directions flip frequently, producing zig-zag oscillation. Momentum adds a velocity term that exponentially averages past gradients:

v_t = \gamma v_{t-1} + \eta g_t

\theta_{t+1} = \theta_t - v_t

. Expanding the recursion as a geometric series

v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}

: older gradients receive weight

\gamma^k

(default

\gamma = 0.9

; after 10 steps the contribution decays to

0.9^{10} \approx 0.35

). Consistent gradient directions accumulate while high-frequency reversals cancel out, so momentum accelerates along the valley and damps oscillation. NAG (Nesterov accelerated gradient) extrapolates one step along the velocity before computing the gradient:

v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})

\theta_{t+1} = \theta_t - v_t

; it is more sensitive to changes in curvature and improves the convex convergence rate from

O(1/t)

(SGD) to

O(1/t^2)

💡Use Cases

the classic baseline for large-scale data-parallel training (ImageNet/CIFAR classification), often still preferred over Adam in CV; interview favorites: why momentum suppresses oscillation, NAG vs vanilla momentum, and how momentum interacts with learning-rate decay.

⚡Key Problems Solved

① oscillation — the velocity lags behind the gradient and averages out high-frequency sign flips, converging much faster on ill-conditioned problems; ② plateaus / small local minima — inertia lets it traverse flat regions and narrow gullies, though too much momentum causes overshoot and rebound around minima, which requires LR scheduling; ③ convergence rate — momentum/NAG improves SGD's

O(1/t)

to NAG's

O(1/t^2)

(optimal constant for first-order methods) at the negligible cost of one extra velocity variable per step.

🎯5 High-Frequency Exam Points

Write the momentum update

v_t = \gamma v_{t-1} + \eta g_t

\theta_{t+1} = \theta_t - v_t

; expand

v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}

and explain how historical gradients decay by

\gamma^k

Why does momentum suppress oscillation and speed up convergence? Explain via the geometric weighted sum: consistent gradients accumulate, high-frequency reversals cancel, and why this helps on contours with large condition number

\kappa

Write the NAG update

v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})

; how does the look-ahead step differ from vanilla momentum, and why does the convex rate improve from

O(1/t)

O(1/t^2)

What does the default

\gamma = 0.9

mean? What problems arise from too large / too small

\gamma

(overshoot vs no memory)? Why is the effective step under constant gradient

\eta/(1-\gamma)

(~10x amplification)?

How does momentum interact with LR scheduling? Why does momentum usually require LR decay (amplified effective step, end-of-training oscillation)? What is the single-learning-rate limitation of momentum vs Adam?

📖 In-depth Guide:📄 optimizer-and-initialization →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "SGD & Momentum"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardSSM & Mamba Next Card →Adam & AdamW

🔗 More Deep Learning Knowledge Cards

Activation Functions Autograd Compute Graph Batch Normalization Classic CNN Architectures