M3-020M3: Deep Learning FoundationsNormalization TechniquesEasy
⚡Flashcard Drill➔Mastery:
Comparing BatchNorm and LayerNorm: Normalization Dimensions and Applicable Regimes
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
BatchNorm normalizes across batch and spatial dimensions per channel (CNNs); LayerNorm normalizes across feature channels per individual sample (Transformers/RNNs).
📌 Key Takeaways
- •BatchNorm: normalizes over for each channel ; couples samples within a batch; fails on small batches
- •LayerNorm: normalizes over hidden channels (or ) for each independent sample; batch-size independent
- •Sequence handling: LayerNorm handles variable-length sequences seamlessly without padding distortion
📐 Mathematical Derivations
Mathematical Formulations:
Let tensor shape be for vision or for sequences.
① Batch Normalization (Ioffe & Szegedy, 2015):
Computes mean and variance across batch and spatial dimensions for each channel : , .
Normalized value: .
② Layer Normalization (Ba, Kiros, Hinton, 2016):
Computes mean and variance across all features/channels for each individual instance at time-step : , .
Normalized value: .
Let tensor shape be for vision or for sequences.
① Batch Normalization (Ioffe & Szegedy, 2015):
Computes mean and variance across batch and spatial dimensions for each channel : , .
Normalized value: .
② Layer Normalization (Ba, Kiros, Hinton, 2016):
Computes mean and variance across all features/channels for each individual instance at time-step : , .
Normalized value: .
🏭 Production Trade-offs
Why NLP uses LayerNorm: In NLP, sequences have variable lengths and batch sizes can vary dynamically down to 1 during inference. BatchNorm over sequences causes zero-padding tokens to corrupt channel statistics and introduces train-test discrepancies. LayerNorm operates completely independently of other samples in the batch.
⚠️ Common Interview Pitfalls
- ✕Using BatchNorm on small batch sizes (), leading to noisy variance estimates and catastrophic training degradation
- ✕Forgetting to call
model.eval()before running inference with BatchNorm, causing predictions to vary depending on test batch composition
🎯 Interviewer Follow-up Inquiries
- ?Why does BatchNorm fail on small batch sizes, and why doesn't LayerNorm suffer from this problem?
- ?How does SyncBatchNorm coordinate cross-GPU statistics in distributed multi-GPU training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚