M3-074M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionHard
⚡Flashcard Drill➔Mastery:

Operators Most Prone to Instability Under Low-Precision (FP16/BF16)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-075 →
📐 Mathematical Definition
risky={exp⁡,log⁡,÷,sum-reduction, θ←θ+ηΔ},ηΔ≪θ\text{risky}=\{\exp,\log,\div,\text{sum-reduction},\ \theta\leftarrow\theta+\eta\Delta\},\quad \eta\Delta\ll\theta
⚡ One-Sentence Core Answer
Operators involving exponentials (Softmax, GELU), logarithms (Cross-Entropy), division (LayerNorm), and massive summations (large-dim matrix multiplications) are most vulnerable.

📌 Key Takeaways

  • •
    Exponentials (exp): Softmax, Sigmoid, and GELU overflow in FP16 when input exceeds 11.0911.09
  • •
    Divisions & Inversions: LayerNorm, RMSNorm, and batch variance calculation amplify small precision errors
  • •
    Accumulation reduction: FP16 matrix multiplications across large hidden dimensions (d=8192d=8192) suffer from accumulator underflow/swamping

📐 Mathematical Derivations

Vulnerability Analysis:
① Transcendental Exponentials: exp⁡(x)\exp(x) overflows in FP16 at x>ln⁡(65504)≈11.09x > \ln(65504) \approx 11.09. In self-attention, if qTk/dk>11.09q^T k / \sqrt{d_k} > 11.09, standard FP16 softmax immediately produces Inf. Fused FP32 accumulation and max-subtraction are mandatory.
② Variance and Standard Deviation: Computing σ2=1d∑(xi−μ)2\sigma^2 = \frac{1}{d} \sum (x_i - \mu)^2. In low precision, catastrophic cancellation occurs when subtracting large near-equal numbers (xi2−μ2x_i^2 - \mu^2), occasionally producing negative variance σ2<0\sigma^2 < 0. Taking σ2\sqrt{\sigma^2} produces NaN.
③ Catastrophic Swamping in Reductions:
When accumulating N=100,000N=100,000 small values into a large sum: St+1=St+xiS_{t+1} = S_t + x_i. When St>211xiS_t > 2^{11} x_i, low mantissa bits in FP16 are rounded off, causing small gradient increments to be completely ignored (swamping).
PyTorch Autocast Defense: torch.cuda.amp.autocast automatically leaves Softmax, LayerNorm, and Cross-Entropy in FP32 while executing large MatMuls and Convs in FP16/BF16.

🏭 Production Trade-offs

Autocast whitelist/blacklist: Always maintain FP32 master weights and execute loss computation, softmax, and normalization in FP32, casting back to FP16/BF16 only for linear/GEMM operations.
⚠️ Common Interview Pitfalls
  • ✕
    Manually casting LayerNorm or Softmax to FP16 inside custom model definitions, bypassing autocast safeguards
  • ✕
    Computing Softmax\text{Softmax} over long sequence lengths in pure FP16 without online FP32 accumulation
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does PyTorch's autocast selectively keep LayerNorm and Softmax in FP32 during mixed precision execution?
  • ?
    What is 'catastrophic cancellation' in floating-point subtraction, and how does Welford's algorithm avoid it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-073: Systematic Troubleshooting Workflow When Training Loss Becomes NaN📋 Back to BankNext →M3-075: Gradient Checkpointing (Activation Rematerialization): Memory vs Compute Trade-offs