M3-057M3: Deep Learning FoundationsGradient Vanishing & ExplosionMedium
⚡Flashcard Drill➔Mastery:
How to Detect and Isolate Gradient Anomalies: Diagnostic Signals and Workflows
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Log layer-wise gradient norms, parameter norms, and update ratios using autograd hooks; diagnose vanishing (monotonic decay), explosion (spikes), and layer imbalances.
📌 Key Takeaways
- •Layer-wise monitoring: register
register_full_backward_hookto log $\|
abla_{W_l} \mathcal{L}\|_2$ per layer - •Four anomaly classes: Vanishing (exponential decay from output to input), Explosion (spikes ), Dead units (zero norm), and Layer Imbalance
- •Update-to-Weight Ratio: healthy updates satisfy
📐 Mathematical Derivations
Diagnostic Framework and Signals:
① Layer-wise Gradient Hook Pattern:
```python
grad_norms = {}
for name, param in model.named_parameters():
if param.requires_grad:
param.register_hook(lambda g, n=name: grad_norms.update({n: g.norm().item()}))
② Anomaly Signatures:
- Vanishing Gradients: Plotting against layer index reveals a steep downward slope where early layers have norms while output layers have norms . Diagnosis: Missing residual connection, incorrect initialization, or saturating activations.
- Exploding Gradients / Loss Spikes: Sudden 2-order-of-magnitude spike in global gradient norm, followed 1–3 steps later by a loss explosion. Diagnosis: Bad data batch or unbounded attention logits.
- Embedding-Output Mismatch: When embeddings are tied with the output projection head (), output gradients often overpower input embedding updates, requiring scale factors ().
① Layer-wise Gradient Hook Pattern:
```python
grad_norms = {}
for name, param in model.named_parameters():
if param.requires_grad:
param.register_hook(lambda g, n=name: grad_norms.update({n: g.norm().item()}))
`② Anomaly Signatures:
- Vanishing Gradients: Plotting against layer index reveals a steep downward slope where early layers have norms while output layers have norms . Diagnosis: Missing residual connection, incorrect initialization, or saturating activations.
- Exploding Gradients / Loss Spikes: Sudden 2-order-of-magnitude spike in global gradient norm, followed 1–3 steps later by a loss explosion. Diagnosis: Bad data batch or unbounded attention logits.
- Embedding-Output Mismatch: When embeddings are tied with the output projection head (), output gradients often overpower input embedding updates, requiring scale factors ().
🏭 Production Trade-offs
Automated debugging tools: Use
torch.autograd.set_detect_anomaly(True) to identify the exact forward operation that generated a NaN gradient, but disable it in production runs due to severe CPU tracking overhead.⚠️ Common Interview Pitfalls
- ✕Only logging the scalar global gradient norm, which completely masks localized layer-wise vanishing in early layers
- ✕Leaving
torch.autograd.set_detect_anomaly(True)enabled during production training, slowing down iterations by
🎯 Interviewer Follow-up Inquiries
- ?Why does a gradient norm spike typically precede a loss spike by 1 to 3 optimization steps?
- ?How does Weight Tying between token embeddings and output projections affect gradient magnitude distribution?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚