Gradient Clipping & NaN Loss Debug

Gradient Clipping & NaN Loss Debugging provides the systematic troubleshooting framework to detect, isolate, and remediate training instabilities, exploding gradients, and numerical divergence in deep neural networks and LLMs; gradient explosion causes parameters to take catastrophic updates into steep non-convex walls, producing arithmetic overflows that cascade into `Loss = NaN`; standard remedies include: 1) Global Norm Gradient Clipping ($g \leftarrow g \cdot \frac{\text{max\_norm}}{\|g\|_2}$ whenever $\|g\|_2 > \text{max\_norm}$, preserving directional vectors); 2) Dynamic Loss Scaling in mixed-precision FP16 (scaling loss by $S=2^{16}$ to shift tiny gradients into representable float exponents while discarding NaN steps); 3) Anomaly tracing with PyTorch `detect_anomaly` and backward hooks.

Large-scale LLM pre-training loss spike debugging, deep recurrent/transformer network stabilization, and FP16/BF16 numerical audits.

Mid-training loss divergence in distributed clusters burns tens of thousands of compute dollars per incident; gradient clipping and loss scaling guarantee resilient, continuous convergence.

Derive Global Norm Gradient Clipping and explain why it preserves gradient direction vectors unlike element-wise value clipping?

Explain the state-machine rules of Dynamic Loss Scaling in FP16 mixed-precision training?

Why does BF16's 8-bit exponent match FP32's dynamic range, eliminating the need for Loss Scalers required by FP16's 5-bit exponent?

List the 5 most common mathematical operations that trigger NaNs in PyTorch computation graphs and their specific fixes?

How to monitor the Grad-Norm-to-Weight-Norm ratio as a vital diagnostic metric for optimizer learning rate health?