M3-073M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionMedium
⚡Flashcard Drill➔Mastery:
Systematic Troubleshooting Workflow When Training Loss Becomes NaN
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Isolate the first NaN occurrence via anomaly detection hooks; audit data inputs for corrupted values, inspect operators for division-by-zero or unmasked logits, and check FP16 overflow.
📌 Key Takeaways
- •First failure isolation: use
torch.autograd.set_detect_anomaly(True)to find the exact operation that produced the first NaN - •Root causes: in normalizations, in CE, attention logit blowups, all-masked attention tokens, unscaled FP16
- •Data sanitization: check for NaNs, Infs, or empty strings in newly loaded data shards
📐 Mathematical Derivations
Systematic Debugging Workflow:
① Step 1: Check Input Data Batches:
Assert
② Step 2: Inspect Numerical Operations:
- Normalization Division: Check if in LayerNorm/RMSNorm is too small () or missing, causing division by zero when variance is zero.
- Softmax Exponents: Ensure logits are stabilized with max-subtraction: .
- Square Root / Log: Check if variance in custom loss became negative due to precision roundoff:
③ Step 3: Audit Precision & Learning Rate:
- If training in FP16, switch temporarily to BF16 or FP32. If NaN disappears, root cause is FP16 exponent overflow ().
- Check gradient norms before NaN; if norms spiked exponentially over 3 steps, learning rate is too high or warmup was insufficient.
① Step 1: Check Input Data Batches:
Assert
torch.isnan(x).any() == False and torch.isinf(x).any() == False. Check for zero-length sequences or attention masks where an entire sequence is masked out with (which causes ).② Step 2: Inspect Numerical Operations:
- Normalization Division: Check if in LayerNorm/RMSNorm is too small () or missing, causing division by zero when variance is zero.
- Softmax Exponents: Ensure logits are stabilized with max-subtraction: .
- Square Root / Log: Check if variance in custom loss became negative due to precision roundoff:
torch.sqrt(torch.clamp(var, min=1e-8)).③ Step 3: Audit Precision & Learning Rate:
- If training in FP16, switch temporarily to BF16 or FP32. If NaN disappears, root cause is FP16 exponent overflow ().
- Check gradient norms before NaN; if norms spiked exponentially over 3 steps, learning rate is too high or warmup was insufficient.
🏭 Production Trade-offs
Automated safeguards: In production distributed pre-training frameworks (Megatron-LM, DeepSpeed), configure automatic batch skipping and checkpoint rollback when loss is non-finite.
⚠️ Common Interview Pitfalls
- ✕Assuming NaNs are always caused by model architecture when are caused by corrupted input data tokens
- ✕Lowering learning rate blindly without verifying whether a normalization layer divided by zero
🎯 Interviewer Follow-up Inquiries
- ?How does an all-padding sequence in masked multi-head attention trigger a NaN in softmax?
- ?How does
torch.autograd.set_detect_anomalytrace the forward origin of a backward NaN?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚