M1-035M1: Mathematics & Statistics FundamentalsNumerical StabilityHard
⚡Flashcard Drill➔Mastery:
Enumerate Common Sources of Numerical Instability in Deep Learning and Their Standard Industrial Mitigations
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Numerical instability stems from vanishing/exploding gradients, floating-point overflow/underflow, matrix near-singularity, and precision loss; mitigated by normalization, gradient clipping, log-domain computation, and mixed precision design.
📌 Key Takeaways
- •Gradient explosion/vanishing: Mitigated via Residual connections, LayerNorm/RMSNorm, proper initialization (Xavier/Kaiming), and Gradient Clipping.
- •Exponent overflow/underflow: Mitigated via Log-Sum-Exp tricks, subtracting max in Softmax, and log1p/expm1 formulations.
- •FP16 dynamic range overflow: Mitigated via dynamic loss scaling, FP32 master weights, or upgrading to BF16/FP8 with scaling.
- •Matrix division by zero: Mitigated via numerical guards (epsilon clamping:
x / (norm + eps)).
📐 Mathematical Derivations
Survey of instability equations: (1) Gradient norm explosion in deep RNNs/Transformers: , projecting gradients back into a bounded hypersphere without changing direction. (2) Epsilon placement in normalization: In RMSNorm, , where (FP32) or prevents division by zero when activations collapse. (3) Activation saturation: Sigmoid derivatives , compounding to ; replaced by non-saturating activations like GELU and SwiGLU.
🏭 Production Trade-offs
System design trade-offs: (1) Epsilon magnitude: Setting too small (e.g. in FP16) results in underflow to zero, causing division by zero anyway; setting it too large (e.g. ) distorts normalization variance. (2) BF16 vs FP16: BF16 trades 3 mantissa bits for 3 exponent bits, sacrificing precision ( relative error) for massive dynamic range (), proving vastly superior for large model pretraining stability.
⚠️ Common Interview Pitfalls
- ✕Placing epsilon outside the square root:
torch.sqrt(var) + epsvstorch.sqrt(var + eps)(the former allowssqrt(0)whose gradient isinf). - ✕Clipping gradients element-wise instead of globally by norm, which alters the directional angle of the gradient vector.
🎯 Interviewer Follow-up Inquiries
- ?Why is
torch.sqrt(var + eps)mathematically safe in backward autodiff whiletorch.sqrt(var) + epsproduces NaN gradients at zero? - ?How does weight decay decouple from adaptive gradients in AdamW to prevent numerical divergence?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚