M3-060M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:
The Loss Spike Phenomenon: Gradient Norm Correlations, Root Causes, and Mitigations
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Loss spikes are sudden 1-to-2 order-of-magnitude error explosions caused by corrupted data batches, outlier activation channels, or attention logit blowups; mitigate via checkpoint rollback and LR annealing.
📌 Key Takeaways
- •Precursor signal: gradient norm spikes sharply 1–3 steps before the loss explodes
- •Root causes: corrupted text sequences, rare token combinations, extreme activation outliers, and FP16 underflow/overflow
- •Mitigations: data deduplication/filtering, global norm clipping, dynamic loss rollback, and QK-Norm
📐 Mathematical Derivations
Mechanism and Dynamics (Chowdhery et al., PaLM 2022; Touvron et al., LLaMA):
During large-scale LLM training, loss curves occasionally experience sudden, catastrophic spikes (e.g., loss jumps from 1.8 to 8.5) and fail to recover or take thousands of steps to re-converge.
- Sequence of Events:
1. A batch containing abnormal data (e.g., repetitive web garbage, extreme token lengths, foreign encoding) enters the network.
2. A few attention logits explode (), driving softmax into near one-hot states and generating massive local gradients.
3. Parameter update moves the model entirely out of the local flat basin into a high-loss barrier ridge.
4. Gradient norms jump by .
- Mitigation Checklist:
- Structural: Use QK-Norm and z-loss (logit penalty) to prevent attention logit growth.
- Data: Strict perplexity-based filtering and deduplication of training corpora.
- Operational: Automated training daemons: detect $\|
abla \mathcal{L}\| > 5\times \text{median}N$ steps.
During large-scale LLM training, loss curves occasionally experience sudden, catastrophic spikes (e.g., loss jumps from 1.8 to 8.5) and fail to recover or take thousands of steps to re-converge.
- Sequence of Events:
1. A batch containing abnormal data (e.g., repetitive web garbage, extreme token lengths, foreign encoding) enters the network.
2. A few attention logits explode (), driving softmax into near one-hot states and generating massive local gradients.
3. Parameter update moves the model entirely out of the local flat basin into a high-loss barrier ridge.
4. Gradient norms jump by .
- Mitigation Checklist:
- Structural: Use QK-Norm and z-loss (logit penalty) to prevent attention logit growth.
- Data: Strict perplexity-based filtering and deduplication of training corpora.
- Operational: Automated training daemons: detect $\|
abla \mathcal{L}\| > 5\times \text{median}N$ steps.
🏭 Production Trade-offs
Rollback vs Continuation: Minor spikes often recover naturally after several dozen steps. However, severe spikes that trigger permanent perplexity degradation must be rolled back to an earlier checkpoint, skipping the offending data shard.
⚠️ Common Interview Pitfalls
- ✕Restarting training from scratch upon encountering a loss spike instead of rolling back 500 steps and skipping the corrupt batch
- ✕Treating loss spikes purely as an optimizer learning rate bug when the root cause is almost always corrupted input tokens
🎯 Interviewer Follow-up Inquiries
- ?How does auxiliary z-loss () prevent logit explosion in output softmax layers?
- ?What automated heuristic rules do LLM training infrastructures use to trigger checkpoint rollbacks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚