M3-056M3: Deep Learning FoundationsGradient Vanishing & ExplosionMedium
⚡Flashcard Drill➔Mastery:
Gradient Noise Scale ($B^*$): Definition and Its Relationship to Optimal Batch Size
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The Gradient Noise Scale () measures the ratio of gradient covariance trace to squared true gradient norm, identifying the critical batch size where data parallelism saturates.
📌 Key Takeaways
- •Definition: , where and
- •Scaling regimes: when batch size , increasing batch size yields linear speedup; when , speedup saturates completely
- •Dynamic evolution: is small at initialization and increases by orders of magnitude as training approaches a flat minimum
📐 Mathematical Derivations
Mathematical Foundation (McCandlish et al., OpenAI 2018; 'An Empirical Model of Large-Batch Training'):
Let be the mini-batch gradient with true expectation and covariance .
For a batch of size , the sample gradient mean has variance .
The expected squared norm of the estimated gradient is: .
Define the Gradient Noise Scale: .
Then: .
- When : Noise dominates. Every new sample in the batch provides independent signal. Compute efficiency is near , and linear batch size scaling holds.
- When : The transition knee. Diminishing returns begin.
- When : Signal dominates. Adding more samples merely reduces an already negligible gradient estimation error, yielding zero algorithmic convergence speedup per step.
Let be the mini-batch gradient with true expectation and covariance .
For a batch of size , the sample gradient mean has variance .
The expected squared norm of the estimated gradient is: .
Define the Gradient Noise Scale: .
Then: .
- When : Noise dominates. Every new sample in the batch provides independent signal. Compute efficiency is near , and linear batch size scaling holds.
- When : The transition knee. Diminishing returns begin.
- When : Signal dominates. Adding more samples merely reduces an already negligible gradient estimation error, yielding zero algorithmic convergence speedup per step.
🏭 Production Trade-offs
Dynamic Batch Sizing Strategy: Because increases by to over the course of training, optimal training pipelines (e.g., GPT-3) gradually ramp batch size from a small initial value ( tokens) to a large mature value ( tokens).
⚠️ Common Interview Pitfalls
- ✕Setting batch size from step 0, wasting GPU compute on redundant, non-informative gradient evaluations
- ✕Assuming remains constant throughout training; it evolves dynamically with model convergence
🎯 Interviewer Follow-up Inquiries
- ?How can the Gradient Noise Scale be measured empirically during training using two different batch sizes?
- ?Why does increase as a model approaches convergence?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚