M3-073M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionMedium
⚡Flashcard Drill➔Mastery:

Systematic Troubleshooting Workflow When Training Loss Becomes NaN

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-074 →
📐 Mathematical Definition
NaN⇐{0/0, ∞/∞, log⁡0, exp⁡(large), neg}\text{NaN}\Leftarrow\{0/0,\ \infty/\infty,\ \log0,\ \exp(\text{large}),\ \sqrt{\text{neg}}\}
⚡ One-Sentence Core Answer
Isolate the first NaN occurrence via anomaly detection hooks; audit data inputs for corrupted values, inspect operators for division-by-zero or unmasked logits, and check FP16 overflow.

📌 Key Takeaways

  • •
    First failure isolation: use torch.autograd.set_detect_anomaly(True) to find the exact operation that produced the first NaN
  • •
    Root causes: 0/00/0 in normalizations, log⁡(0)=−∞\log(0) = -\infty in CE, attention logit blowups, all-masked attention tokens, unscaled FP16
  • •
    Data sanitization: check for NaNs, Infs, or empty strings in newly loaded data shards

📐 Mathematical Derivations

Systematic Debugging Workflow:
① Step 1: Check Input Data Batches:
Assert torch.isnan(x).any() == False and torch.isinf(x).any() == False. Check for zero-length sequences or attention masks where an entire sequence is masked out with −∞-\infty (which causes softmax(−∞)=0/0=NaN\text{softmax}(-\infty) = 0/0 = \text{NaN}).
② Step 2: Inspect Numerical Operations:
- Normalization Division: Check if ϵ\epsilon in LayerNorm/RMSNorm is too small (<10−8< 10^{-8}) or missing, causing division by zero when variance is zero.
- Softmax Exponents: Ensure logits are stabilized with max-subtraction: softmax(z−max⁡(z))\text{softmax}(z - \max(z)).
- Square Root / Log: Check if variance in custom loss became negative due to precision roundoff: torch.sqrt(torch.clamp(var, min=1e-8)).
③ Step 3: Audit Precision & Learning Rate:
- If training in FP16, switch temporarily to BF16 or FP32. If NaN disappears, root cause is FP16 exponent overflow (>65504> 65504).
- Check gradient norms before NaN; if norms spiked exponentially over 3 steps, learning rate is too high or warmup was insufficient.

🏭 Production Trade-offs

Automated safeguards: In production distributed pre-training frameworks (Megatron-LM, DeepSpeed), configure automatic batch skipping and checkpoint rollback when loss is non-finite.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming NaNs are always caused by model architecture when >70%>70\% are caused by corrupted input data tokens
  • ✕
    Lowering learning rate blindly without verifying whether a normalization layer divided by zero
🎯 Interviewer Follow-up Inquiries
  • ?
    How does an all-padding sequence in masked multi-head attention trigger a 0/00/0 NaN in softmax?
  • ?
    How does torch.autograd.set_detect_anomaly trace the forward origin of a backward NaN?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-072: Loss Scaling Mechanism and Dynamic Adjustment in Mixed Precision📋 Back to BankNext →M3-074: Operators Most Prone to Instability Under Low-Precision (FP16/BF16)