M3-072M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionMedium
⚡Flashcard Drill➔Mastery:

Loss Scaling Mechanism and Dynamic Adjustment in Mixed Precision

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-073 →
📐 Mathematical Definition
L~=s⋅L;g~=∇L~=s⋅g;θ←θ−η g~/s\tilde{\mathcal{L}}=s\cdot\mathcal{L};\qquad \tilde g=\nabla\tilde{\mathcal{L}}=s\cdot g;\qquad \theta\leftarrow\theta-\eta\,\tilde g/s
⚡ One-Sentence Core Answer
Loss scaling multiplies the forward loss by scale SS to push small backpropagated gradients into FP16 representable range, unscaling before the optimizer update.

📌 Key Takeaways

  • •
    Underflow problem: over 80% of backpropagated gradients in deep networks are smaller than FP16 minimum (6.1×10−56.1 \times 10^{-5})
  • •
    Mechanism: gscaled=∇θ(S⋅L)=S⋅gg_{\text{scaled}} = \nabla_\theta (S \cdot \mathcal{L}) = S \cdot g; divides by SS in FP32 before weight update
  • •
    Dynamic heuristic: start with large scale SS; if Inf/NaN appears, skip update and halve SS; if NN clean steps, double SS

📐 Mathematical Derivations

Mathematical Formulations (Micikevicius et al., ICLR 2018; Mixed Precision Training):
By the chain rule, scaling loss by constant SS scales all gradients linearly:
∇θ(S⋅L)=S⋅∇θL=S⋅g\nabla_\theta (S \cdot \mathcal{L}) = S \cdot \nabla_\theta \mathcal{L} = S \cdot g.
- This shifts the entire gradient magnitude histogram to the right by log⁡2(S)\log_2(S) bits, rescuing gradients from the underflow zone [0,2−14][0, 2^{-14}].
- Dynamic Loss Scaling Algorithm:
1. Initialize scale S=216=65536S = 2^{16} = 65536, growth interval N=2000N = 2000.
2. Forward pass in FP16; compute scaled loss Lscaled=S⋅L\mathcal{L}_{\text{scaled}} = S \cdot \mathcal{L}.
3. Backward pass in FP16; produces scaled gradients S⋅gS \cdot g.
4. Inspect all gradients for Inf or NaN.
- If Inf/NaN detected (Overflow): Discard the entire batch update (do not call optimizer.step()), reduce scale: S←max⁡(1,S/2)S \leftarrow \max(1, S / 2), and reset consecutive successful step counter c=0c = 0.
- If clean: Unscale gradients in FP32 master weights: gtrue=1S(S⋅g)g_{\text{true}} = \frac{1}{S} (S \cdot g). Clip gradients and execute optimizer.step(). Increment counter c←c+1c \leftarrow c + 1.
- If c==Nc == N: Double the scale: S←2SS \leftarrow 2 S, and reset c=0c = 0.

🏭 Production Trade-offs

Master Weights Requirement: Parameters must be stored as FP32 master copies. Optimizer updates are applied to FP32 master weights, which are then cast down to FP16/BF16 for the subsequent forward pass.
⚠️ Common Interview Pitfalls
  • ✕
    Clipping gradients before unscaling them, which multiplies the clipping threshold by SS and nullifies clipping
  • ✕
    Executing optimizer step when gradients contain Inf, permanently corrupting model weights with NaNs
🎯 Interviewer Follow-up Inquiries
  • ?
    Why must gradient unscaling and gradient clipping be executed in a specific sequential order?
  • ?
    Why do master weights need to be maintained in full FP32 precision during mixed precision training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-071: Floating-Point Precisions in Deep Learning: FP16, BF16, and FP32 Trade-offs📋 Back to BankNext →M3-073: Systematic Troubleshooting Workflow When Training Loss Becomes NaN