M3-074M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionHard
⚡Flashcard Drill➔Mastery:
Operators Most Prone to Instability Under Low-Precision (FP16/BF16)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Operators involving exponentials (Softmax, GELU), logarithms (Cross-Entropy), division (LayerNorm), and massive summations (large-dim matrix multiplications) are most vulnerable.
📌 Key Takeaways
- •Exponentials (exp): Softmax, Sigmoid, and GELU overflow in FP16 when input exceeds
- •Divisions & Inversions: LayerNorm, RMSNorm, and batch variance calculation amplify small precision errors
- •Accumulation reduction: FP16 matrix multiplications across large hidden dimensions () suffer from accumulator underflow/swamping
📐 Mathematical Derivations
Vulnerability Analysis:
① Transcendental Exponentials: overflows in FP16 at . In self-attention, if , standard FP16 softmax immediately produces
② Variance and Standard Deviation: Computing . In low precision, catastrophic cancellation occurs when subtracting large near-equal numbers (), occasionally producing negative variance . Taking produces
③ Catastrophic Swamping in Reductions:
When accumulating small values into a large sum: . When , low mantissa bits in FP16 are rounded off, causing small gradient increments to be completely ignored (swamping).
PyTorch Autocast Defense:
① Transcendental Exponentials: overflows in FP16 at . In self-attention, if , standard FP16 softmax immediately produces
Inf. Fused FP32 accumulation and max-subtraction are mandatory.② Variance and Standard Deviation: Computing . In low precision, catastrophic cancellation occurs when subtracting large near-equal numbers (), occasionally producing negative variance . Taking produces
NaN.③ Catastrophic Swamping in Reductions:
When accumulating small values into a large sum: . When , low mantissa bits in FP16 are rounded off, causing small gradient increments to be completely ignored (swamping).
PyTorch Autocast Defense:
torch.cuda.amp.autocast automatically leaves Softmax, LayerNorm, and Cross-Entropy in FP32 while executing large MatMuls and Convs in FP16/BF16.🏭 Production Trade-offs
Autocast whitelist/blacklist: Always maintain FP32 master weights and execute loss computation, softmax, and normalization in FP32, casting back to FP16/BF16 only for linear/GEMM operations.
⚠️ Common Interview Pitfalls
- ✕Manually casting LayerNorm or Softmax to FP16 inside custom model definitions, bypassing autocast safeguards
- ✕Computing over long sequence lengths in pure FP16 without online FP32 accumulation
🎯 Interviewer Follow-up Inquiries
- ?Why does PyTorch's
autocastselectively keep LayerNorm and Softmax in FP32 during mixed precision execution? - ?What is 'catastrophic cancellation' in floating-point subtraction, and how does Welford's algorithm avoid it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚