M1-035M1: Mathematics & Statistics FundamentalsNumerical StabilityHard
⚡Flashcard Drill➔Mastery:

Enumerate Common Sources of Numerical Instability in Deep Learning and Their Standard Industrial Mitigations

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-036 →
📐 Mathematical Definition
缓解: log⁡-sum-exp, ϵ 平滑, clipping, BF16, LN/RMSNorm\text{缓解}:\ \log\text{-sum-exp},\ \epsilon\ \text{平滑},\ \text{clipping},\ \text{BF16},\ \text{LN/RMSNorm}
⚡ One-Sentence Core Answer
Numerical instability stems from vanishing/exploding gradients, floating-point overflow/underflow, matrix near-singularity, and precision loss; mitigated by normalization, gradient clipping, log-domain computation, and mixed precision design.

📌 Key Takeaways

  • •
    Gradient explosion/vanishing: Mitigated via Residual connections, LayerNorm/RMSNorm, proper initialization (Xavier/Kaiming), and Gradient Clipping.
  • •
    Exponent overflow/underflow: Mitigated via Log-Sum-Exp tricks, subtracting max in Softmax, and log1p/expm1 formulations.
  • •
    FP16 dynamic range overflow: Mitigated via dynamic loss scaling, FP32 master weights, or upgrading to BF16/FP8 with scaling.
  • •
    Matrix division by zero: Mitigated via numerical guards (epsilon clamping: x / (norm + eps)).

📐 Mathematical Derivations

Survey of instability equations: (1) Gradient norm explosion in deep RNNs/Transformers: g←g⋅min⁡(1,clip_norm∥g∥2)g \leftarrow g \cdot \min\left(1, \frac{\text{clip\_norm}}{\|g\|_2}\right), projecting gradients back into a bounded hypersphere without changing direction. (2) Epsilon placement in normalization: In RMSNorm, RMS(x)=1d∑i=1dxi2+ϵ\text{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}, where ϵ=10−5\epsilon = 10^{-5} (FP32) or 10−610^{-6} prevents division by zero when activations collapse. (3) Activation saturation: Sigmoid derivatives σ′(z)=σ(z)(1−σ(z))≤0.25\sigma'(z) = \sigma(z)(1-\sigma(z)) \le 0.25, compounding to 0.25L→00.25^L \to 0; replaced by non-saturating activations like GELU and SwiGLU.

🏭 Production Trade-offs

System design trade-offs: (1) Epsilon magnitude: Setting ϵ\epsilon too small (e.g. 10−1210^{-12} in FP16) results in underflow to zero, causing division by zero anyway; setting it too large (e.g. 10−210^{-2}) distorts normalization variance. (2) BF16 vs FP16: BF16 trades 3 mantissa bits for 3 exponent bits, sacrificing precision (10−310^{-3} relative error) for massive dynamic range (103810^{38}), proving vastly superior for large model pretraining stability.
⚠️ Common Interview Pitfalls
  • ✕
    Placing epsilon outside the square root: torch.sqrt(var) + eps vs torch.sqrt(var + eps) (the former allows sqrt(0) whose gradient 120\frac{1}{2\sqrt{0}} is inf).
  • ✕
    Clipping gradients element-wise instead of globally by norm, which alters the directional angle of the gradient vector.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is torch.sqrt(var + eps) mathematically safe in backward autodiff while torch.sqrt(var) + eps produces NaN gradients at zero?
  • ?
    How does weight decay decouple from adaptive gradients in AdamW to prevent numerical divergence?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-034: Explain Why Numerically Stable Implementations Use log1p(exp(-|z|)) for Binary Cross-Entropy (BCEWithLogits)📋 Back to BankNext →M1-036: Define Maximum Likelihood Estimation (MLE) with a Complete End-to-End Derivation