M3-072M3: Deep Learning FoundationsTraining Stability & Mixed PrecisionMedium
⚡Flashcard Drill➔Mastery:
Loss Scaling Mechanism and Dynamic Adjustment in Mixed Precision
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Loss scaling multiplies the forward loss by scale to push small backpropagated gradients into FP16 representable range, unscaling before the optimizer update.
📌 Key Takeaways
- •Underflow problem: over 80% of backpropagated gradients in deep networks are smaller than FP16 minimum ()
- •Mechanism: ; divides by in FP32 before weight update
- •Dynamic heuristic: start with large scale ; if
Inf/NaNappears, skip update and halve ; if clean steps, double
📐 Mathematical Derivations
Mathematical Formulations (Micikevicius et al., ICLR 2018; Mixed Precision Training):
By the chain rule, scaling loss by constant scales all gradients linearly:
.
- This shifts the entire gradient magnitude histogram to the right by bits, rescuing gradients from the underflow zone .
- Dynamic Loss Scaling Algorithm:
1. Initialize scale , growth interval .
2. Forward pass in FP16; compute scaled loss .
3. Backward pass in FP16; produces scaled gradients .
4. Inspect all gradients for
- If Inf/NaN detected (Overflow): Discard the entire batch update (do not call
- If clean: Unscale gradients in FP32 master weights: . Clip gradients and execute
- If : Double the scale: , and reset .
By the chain rule, scaling loss by constant scales all gradients linearly:
.
- This shifts the entire gradient magnitude histogram to the right by bits, rescuing gradients from the underflow zone .
- Dynamic Loss Scaling Algorithm:
1. Initialize scale , growth interval .
2. Forward pass in FP16; compute scaled loss .
3. Backward pass in FP16; produces scaled gradients .
4. Inspect all gradients for
Inf or NaN.- If Inf/NaN detected (Overflow): Discard the entire batch update (do not call
optimizer.step()), reduce scale: , and reset consecutive successful step counter .- If clean: Unscale gradients in FP32 master weights: . Clip gradients and execute
optimizer.step(). Increment counter .- If : Double the scale: , and reset .
🏭 Production Trade-offs
Master Weights Requirement: Parameters must be stored as FP32 master copies. Optimizer updates are applied to FP32 master weights, which are then cast down to FP16/BF16 for the subsequent forward pass.
⚠️ Common Interview Pitfalls
- ✕Clipping gradients before unscaling them, which multiplies the clipping threshold by and nullifies clipping
- ✕Executing optimizer step when gradients contain
Inf, permanently corrupting model weights with NaNs
🎯 Interviewer Follow-up Inquiries
- ?Why must gradient unscaling and gradient clipping be executed in a specific sequential order?
- ?Why do master weights need to be maintained in full FP32 precision during mixed precision training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚