M3-042M3: Deep Learning FoundationsLearning Rate SchedulesMedium
⚡Flashcard Drill➔Mastery:
Relationship Between Learning Rate and Batch Size: Linear vs Square-Root Scaling
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The Linear Scaling Rule scales learning rate proportionally with batch size (); when gradient noise is high or under adaptive optimizers, Square-Root scaling () applies.
📌 Key Takeaways
- •Linear Scaling Rule (Goyal et al., 2017): when multiplying batch size by , multiply learning rate by ()
- •Warmup requirement: large batch sizes require gradual warmup to prevent initial instability
- •Critical batch size: beyond a model-specific threshold , linear scaling breaks down due to gradient noise saturation
📐 Mathematical Derivations
Mathematical Justification (Linear Scaling):
Consider iterations of SGD with batch size vs 1 iteration with batch size .
- steps with small batch : .
Assuming parameter drift is small over steps (): .
- 1 step with large batch and scaled learning rate :
.
Equating the two updates yields: .
Square-Root Scaling (Krizhevsky, 2014): Under random walk diffusion dynamics where gradient noise covariance scales as , keeping the parameter diffusion rate constant requires .
Consider iterations of SGD with batch size vs 1 iteration with batch size .
- steps with small batch : .
Assuming parameter drift is small over steps (): .
- 1 step with large batch and scaled learning rate :
.
Equating the two updates yields: .
Square-Root Scaling (Krizhevsky, 2014): Under random walk diffusion dynamics where gradient noise covariance scales as , keeping the parameter diffusion rate constant requires .
🏭 Production Trade-offs
Critical Batch Size Bound: McCandlish et al. (OpenAI, 2018) showed that gradient noise scale defines the boundary. For , linear scaling holds perfectly. For , adding more samples provides diminishing gradient information, and linear scaling causes divergence.
⚠️ Common Interview Pitfalls
- ✕Scaling batch size by without scaling learning rate or adding warmup, causing training throughput to slow by
- ✕Scaling learning rate linearly far past the critical batch size , triggering catastrophic divergence
🎯 Interviewer Follow-up Inquiries
- ?How does OpenAI's gradient noise scale determine the maximum parallelizable batch size?
- ?Why does AdamW typically follow square-root or sub-linear scaling rather than pure linear scaling?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚