M3-003M3: Deep Learning FoundationsBackprop & AutodiffMedium
⚡Flashcard Drill➔Mastery:
Gradient Accumulation: Mathematical Equivalence and Implementation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
It computes gradients over micro-batches and averages them before calling optimizer step, exactly matching the mathematical gradient of effective batch size .
📌 Key Takeaways
- •Mathematical equivalence:
- •Memory benefit: fits large effective batch sizes into constrained GPU VRAM without out-of-memory errors
- •Discrepancy nuances: BatchNorm running statistics and dropout RNG states differ slightly from a true large batch
📐 Mathematical Derivations
Mathematical Equivalence: Let total objective be . The gradient is:
.
Implementation Loop:
```python
for k, (x, y) in enumerate(micro_batches):
loss = criterion(model(x), y) / K # Scale loss by accumulation steps
loss.backward() # Gradients accumulate in .grad
if (k + 1) % K == 0:
optimizer.step()
optimizer.zero_grad()
Because PyTorch accumulates gradients via tensor addition (), dividing each micro-batch loss by ensures the accumulated gradient matches the true mean.
.
Implementation Loop:
```python
for k, (x, y) in enumerate(micro_batches):
loss = criterion(model(x), y) / K # Scale loss by accumulation steps
loss.backward() # Gradients accumulate in .grad
if (k + 1) % K == 0:
optimizer.step()
optimizer.zero_grad()
`Because PyTorch accumulates gradients via tensor addition (), dividing each micro-batch loss by ensures the accumulated gradient matches the true mean.
🏭 Production Trade-offs
System design trade-offs: Gradient accumulation trades wall-clock training time for memory. It reduces distributed communication overhead because all-reduce is only triggered once every micro-steps. Note that BatchNorm computes running mean/var per micro-batch, causing slight distribution divergence compared to a true large batch.
⚠️ Common Interview Pitfalls
- ✕Forgetting to divide loss by before
loss.backward(), causing effective learning rate to be multiplied by - ✕Calling
optimizer.zero_grad()inside the micro-batch loop instead of every steps, erasing accumulated gradients
🎯 Interviewer Follow-up Inquiries
- ?Why must gradient synchronization across distributed GPUs (DDP) be disabled during accumulation steps via
no_sync()? - ?How does gradient accumulation affect Batch Normalization compared to Layer Normalization?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚