Vanishing/Exploding Gradients

Vanishing/exploding gradients are the core obstacle to training deep nets and RNNs, rooted in the chain product of Jacobians along depth: $\frac{\partial L}{\partial h_0}=\prod_{t=1}^{T}\frac{\partial h_t}{\partial h_{t-1}}$, whose norm is bounded by $\prod_{t}\Vert J_t\Vert\le\gamma^T$ — if each layer's spectral norm $\gamma<1$ the product decays exponentially to 0 (vanishing); if $\gamma>1$ it explodes. Numeric intuition: scaling by $0.9$ per step gives $0.9^{50}\approx0.005$ after 50 steps (gradient shrinks to ~1/200); scaling by $1.1$ gives $1.1^{50}\approx117$ (overflow). Sigmoid's peak derivative 0.25 (<1) is a classic “vanishing amplifier”. Fixes: ① weight initialization — Xavier preserves forward/backward variance with $\frac{2}{n_{in}+n_{out}}$, Kaiming corrects to $\frac{2}{n_{in}}$ for ReLU; ② residual connections $y=F(x)+x$ give an identity shortcut for gradients; ③ normalization (BN/LN/RMSNorm) stabilizes activation scales; ④ the ReLU family keeps a constant gradient of 1; ⑤ gradient clipping handles explosion, LSTM's additive cell state eases long-term dependencies.

interview staples — “why won't my deep net train?”, “why do RNNs struggle with long dependencies?”, “what does $0.9^{50}\approx0.005$ imply?”; you are expected to chain together ResNet residuals, Xavier/Kaiming init and BatchNorm and explain why each alone is insufficient.

makes hundreds-layer nets (ResNet-152, LLMs) actually trainable — residuals let gradients flow losslessly through the identity shortcut (shallower effective depth, deeper trainable depth); initialization anchors each layer's output variance to the input scale so the product norm stays near $\gamma\approx1$; normalization stabilizes activation scales and Jacobians. Together they break the degradation that appears at ~20 plain layers (training error stops decreasing) and scale nets to 1000+ layers; RNN long-term dependency windows grow from a few steps to dozens.

Test Your Knowledge: Practice Questions for "Vanishing/Exploding Gradients"

Single choice pitfall questions with instant feedback and mistake tracking.