M3-042M3: Deep Learning FoundationsLearning Rate SchedulesMedium
⚡Flashcard Drill➔Mastery:

Relationship Between Learning Rate and Batch Size: Linear vs Square-Root Scaling

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-043 →
📐 Mathematical Definition
η∝B (linear);η∝B (sqrt);Bcrit≈tr(Σ)g⊤g\eta\propto B\ (\text{linear});\qquad \eta\propto\sqrt{B}\ (\text{sqrt});\qquad B_{\text{crit}}\approx\frac{\text{tr}(\Sigma)}{g^{\top}g}
⚡ One-Sentence Core Answer
The Linear Scaling Rule scales learning rate proportionally with batch size (η∝B\eta \propto B); when gradient noise is high or under adaptive optimizers, Square-Root scaling (η∝B\eta \propto \sqrt{B}) applies.

📌 Key Takeaways

  • •
    Linear Scaling Rule (Goyal et al., 2017): when multiplying batch size by kk, multiply learning rate by kk (η′=kη\eta' = k \eta)
  • •
    Warmup requirement: large batch sizes require gradual warmup to prevent initial instability
  • •
    Critical batch size: beyond a model-specific threshold B∗B^*, linear scaling breaks down due to gradient noise saturation

📐 Mathematical Derivations

Mathematical Justification (Linear Scaling):
Consider kk iterations of SGD with batch size BB vs 1 iteration with batch size kBk B.
- kk steps with small batch BB: θt+k=θt−η∑j=0k−11B∑i=1B∇ℓ(xj,i;θt+j)\theta_{t+k} = \theta_t - \eta \sum_{j=0}^{k-1} \frac{1}{B} \sum_{i=1}^B \nabla \ell(x_{j, i}; \theta_{t+j}).
Assuming parameter drift is small over kk steps (θt+j≈θt\theta_{t+j} \approx \theta_t): θt+k≈θt−η∑j=0k−11B∑i=1B∇ℓ(xj,i;θt)\theta_{t+k} \approx \theta_t - \eta \sum_{j=0}^{k-1} \frac{1}{B} \sum_{i=1}^B \nabla \ell(x_{j, i}; \theta_t).
- 1 step with large batch kBk B and scaled learning rate η^\hat{\eta}:
θ^t+1=θt−η^1kB∑j=0k−1∑i=1B∇ℓ(xj,i;θt)\hat{\theta}_{t+1} = \theta_t - \hat{\eta} \frac{1}{k B} \sum_{j=0}^{k-1} \sum_{i=1}^B \nabla \ell(x_{j, i}; \theta_t).
Equating the two updates yields: η^1kB=η1B  ⟹  η^=kη\hat{\eta} \frac{1}{k B} = \eta \frac{1}{B} \implies \hat{\eta} = k \eta.
Square-Root Scaling (Krizhevsky, 2014): Under random walk diffusion dynamics where gradient noise covariance scales as Var∝1/B\text{Var} \propto 1/B, keeping the parameter diffusion rate constant requires η∝B\eta \propto \sqrt{B}.

🏭 Production Trade-offs

Critical Batch Size Bound: McCandlish et al. (OpenAI, 2018) showed that gradient noise scale B∗=tr(Σ)∥g∥2B^* = \frac{\text{tr}(\Sigma)}{\|g\|^2} defines the boundary. For B≪B∗B \ll B^*, linear scaling holds perfectly. For B≫B∗B \gg B^*, adding more samples provides diminishing gradient information, and linear scaling causes divergence.
⚠️ Common Interview Pitfalls
  • ✕
    Scaling batch size by 8×8\times without scaling learning rate or adding warmup, causing training throughput to slow by 8×8\times
  • ✕
    Scaling learning rate linearly far past the critical batch size B∗B^*, triggering catastrophic divergence
🎯 Interviewer Follow-up Inquiries
  • ?
    How does OpenAI's gradient noise scale B∗B^* determine the maximum parallelizable batch size?
  • ?
    Why does AdamW typically follow square-root or sub-linear scaling rather than pure linear scaling?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-041: Comparing Learning Rate Schedules: Cosine, Step, Linear, and WSD📋 Back to BankNext →M3-043: Alternative Motivation for Warmup: Adam's Second-Moment Variance Collapse