Optimizer Convergence & Momentum

Optimizer Convergence Dynamics & Momentum Selection Rules establishes the algorithmic decision matrix choosing between first-order momentum and adaptive learning rate optimizers based on model topology (CNNs, Transformers, Sparse Embeddings); the 3 dominant families are: 1) SGD with Momentum / Nesterov Accelerated Gradient (NAG), dominating Computer Vision due to flat minima generalization; 2) AdamW (Decoupled Weight Decay Adam: updating $\theta_{t+1} = \theta_t - \eta (\frac{m_t}{\sqrt{v_t} + \epsilon} + \lambda \theta_t)$ by isolating weight decay from adaptive gradient scaling), serving as the universal standard for Transformers and LLMs; 3) AdaGrad / FTRL, optimizing sparse categorical embeddings in ad ranking.

LLM pre-training/fine-tuning, computer vision network training, and debugging training divergence or loss oscillations.

Original Adam coupled L2 regularization into gradient updates, causing adaptive learning rate scaling to suppress weight decay on frequent tokens and degrading generalization; AdamW decouples weight decay, achieving fast convergence and superior out-of-distribution generalization.

Derive the mathematical difference between original Adam and AdamW in weight decay updates, explaining why L2 regularization fails in Adam?

Derive the initial step bias correction formulas $\hat{m}_t = \frac{m_t}{1 - \beta_1^t}$ and $\hat{v}_t = \frac{v_t}{1 - \beta_2^t}$ in Adam/AdamW?

Why does SGD with Momentum often generalize better to flat minima in Computer Vision compared to adaptive optimizers?

Explain memory savings of Adafactor (low-rank factorization) and the sign-gradient operation of the Lion optimizer?

How to tune $\beta_1, \beta_2,$ and $\epsilon$ to stabilize training against loss spikes and gradient oscillations in mixed precision?