Gradient Clipping

Gradient clipping bounds the gradient norm before the optimizer step to prevent exploding gradients. Two approaches: ① per-parameter (value) clipping $g_i \leftarrow \text{clip}(g_i, -c, c)$, i.e. $g_i \leftarrow g_i \cdot \min\left(1, \frac{c}{|g_i|}\right)$ — it only bounds individual component magnitudes, scales different components by different factors (distorting the direction), and cannot bound the global norm; ② global-norm clipping: compute the total norm $G = \Vert g \Vert_2 = \sqrt{\sum_i g_i^2}$ (per-layer normalized variant $G = \sqrt{\sum_l \Vert g_l\Vert_2^2 / n_l}$ with $n_l$ = parameter count of layer $l$, so large layers don't dominate), and when $G > c$ scale uniformly $g \leftarrow g \cdot \frac{c}{G}$ — direction preserved, step norm hard-bounded at $c$ (equivalent to projecting onto a ball of radius $c$; PyTorch clip_grad_norm_ commonly uses $c = 1.0$). When to clip: after gradient synchronization (DDP All-Reduce) and before the optimizer step; in mixed precision, clip the gradients of the FP32 master weights (clipping low-precision gradients loses precision). Loss-spike (NaN/Inf) handling: standard procedure when loss suddenly jumps to NaN/Inf — ① diagnose: LR too large? anomalous data samples (corrupted images / missing labels)? gradient overflow? ② if the gradient contains NaN/Inf, skip this optimizer step — NaN gradients corrupt the FP32 master weights and poison all subsequent training; ③ in AMP, check Loss Scaling first: FP16 forward/backward overflow → Inf gradients → halve the dynamic loss scale ($S \leftarrow S/2$) and skip/replay the step; after several overflow-free steps, double $S$; ④ roll back to the last good checkpoint, lower the LR, inspect the data pipeline, tighten clipping if needed; ⑤ monitor long-term: track the grad-norm curve (a loss explosion is preceded by a persistently rising gradient norm) and the loss curve, and auto-pause training on NaN detection.

standard protection for RNN/LSTM and long-sequence Transformer training (chain-multiplied long-range gradients explode most easily), and a routine component of large-batch and mixed-precision training; interview favorites: why global norm instead of per-parameter clipping, how to handle loss spikes, and the ordering of clipping vs AMP/DDP.

exploding gradients can throw parameters out of the normal region in one step (loss → NaN, training ruined); clipping bounds the worst-case step so training continues through pathological moments. Compared with per-parameter clipping, global-norm clipping preserves direction and bounds the norm, making it the engineering default. Note clipping treats the symptom: if it triggers frequently (the scale $c/G$ is often much less than 1), the LR is too large or the model/data have real problems — lower the LR or fix the data instead of relying on the clip.

Test Your Knowledge: Practice Questions for "Gradient Clipping"

Single choice pitfall questions with instant feedback and mistake tracking.