SGD & Momentum

Plain SGD updates along the negative instantaneous gradient: $\theta_{t+1} = \theta_t - \eta g_t$ with $g_t = \nabla_\theta L(\theta_t)$; on ill-conditioned (elliptical) contours with large condition number $\kappa = \lambda_{\max}/\lambda_{\min}$, gradient directions flip frequently, producing zig-zag oscillation. Momentum adds a velocity term that exponentially averages past gradients: $v_t = \gamma v_{t-1} + \eta g_t$, $\theta_{t+1} = \theta_t - v_t$. Expanding the recursion as a geometric series $v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}$: older gradients receive weight $\gamma^k$ (default $\gamma = 0.9$; after 10 steps the contribution decays to $0.9^{10} \approx 0.35$). Consistent gradient directions accumulate while high-frequency reversals cancel out, so momentum accelerates along the valley and damps oscillation. NAG (Nesterov accelerated gradient) extrapolates one step along the velocity before computing the gradient: $v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})$, $\theta_{t+1} = \theta_t - v_t$; it is more sensitive to changes in curvature and improves the convex convergence rate from $O(1/t)$ (SGD) to $O(1/t^2)$.

the classic baseline for large-scale data-parallel training (ImageNet/CIFAR classification), often still preferred over Adam in CV; interview favorites: why momentum suppresses oscillation, NAG vs vanilla momentum, and how momentum interacts with learning-rate decay.

① oscillation — the velocity lags behind the gradient and averages out high-frequency sign flips, converging much faster on ill-conditioned problems; ② plateaus / small local minima — inertia lets it traverse flat regions and narrow gullies, though too much momentum causes overshoot and rebound around minima, which requires LR scheduling; ③ convergence rate — momentum/NAG improves SGD's $O(1/t)$ to NAG's $O(1/t^2)$ (optimal constant for first-order methods) at the negligible cost of one extra velocity variable per step.

Write the momentum update $v_t = \gamma v_{t-1} + \eta g_t$, $\theta_{t+1} = \theta_t - v_t$; expand $v_t = \eta\sum_{k=0}^{t}\gamma^k g_{t-k}$ and explain how historical gradients decay by $\gamma^k$.

Why does momentum suppress oscillation and speed up convergence? Explain via the geometric weighted sum: consistent gradients accumulate, high-frequency reversals cancel, and why this helps on contours with large condition number $\kappa$.

Write the NAG update $v_t = \gamma v_{t-1} + \eta\nabla_\theta L(\theta_t - \gamma v_{t-1})$; how does the look-ahead step differ from vanilla momentum, and why does the convex rate improve from $O(1/t)$ to $O(1/t^2)$?

What does the default $\gamma = 0.9$ mean? What problems arise from too large / too small $\gamma$ (overshoot vs no memory)? Why is the effective step under constant gradient $\eta/(1-\gamma)$ (~10x amplification)?

How does momentum interact with LR scheduling? Why does momentum usually require LR decay (amplified effective step, end-of-training oscillation)? What is the single-learning-rate limitation of momentum vs Adam?