M3-020M3: Deep Learning FoundationsNormalization TechniquesEasy
⚡Flashcard Drill➔Mastery:

Comparing BatchNorm and LayerNorm: Normalization Dimensions and Applicable Regimes

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-021 →
📐 Mathematical Definition
BN: x−μBσB2+ϵ;LN: x−μσ2+ϵ\text{BN}:\ \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}};\qquad \text{LN}:\ \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}
⚡ One-Sentence Core Answer
BatchNorm normalizes across batch and spatial dimensions per channel (CNNs); LayerNorm normalizes across feature channels per individual sample (Transformers/RNNs).

📌 Key Takeaways

  • •
    BatchNorm: normalizes over (N,H,W)(N, H, W) for each channel CC; couples samples within a batch; fails on small batches
  • •
    LayerNorm: normalizes over hidden channels CC (or DD) for each independent sample; batch-size independent
  • •
    Sequence handling: LayerNorm handles variable-length sequences seamlessly without padding distortion

📐 Mathematical Derivations

Mathematical Formulations:
Let tensor shape be (N,C,H,W)(N, C, H, W) for vision or (N,L,D)(N, L, D) for sequences.
① Batch Normalization (Ioffe & Szegedy, 2015):
Computes mean and variance across batch NN and spatial dimensions H,WH, W for each channel c∈[1,C]c \in [1, C]: μc=1NHW∑n=1N∑h=1H∑w=1Wxn,c,h,w\mu_c = \frac{1}{N H W} \sum_{n=1}^N \sum_{h=1}^H \sum_{w=1}^W x_{n, c, h, w}, σc2=1NHW∑(x−μc)2\sigma_c^2 = \frac{1}{N H W} \sum (x - \mu_c)^2.
Normalized value: x^n,c,h,w=x−μcσc2+ϵγc+βc\hat{x}_{n, c, h, w} = \frac{x - \mu_c}{\sqrt{\sigma_c^2 + \epsilon}} \gamma_c + \beta_c.
② Layer Normalization (Ba, Kiros, Hinton, 2016):
Computes mean and variance across all features/channels DD for each individual instance nn at time-step tt: μn,t=1D∑d=1Dxn,t,d\mu_{n, t} = \frac{1}{D} \sum_{d=1}^D x_{n, t, d}, σn,t2=1D∑d=1D(xn,t,d−μn,t)2\sigma_{n, t}^2 = \frac{1}{D} \sum_{d=1}^D (x_{n, t, d} - \mu_{n, t})^2.
Normalized value: x^n,t,d=x−μn,tσn,t2+ϵγd+βd\hat{x}_{n, t, d} = \frac{x - \mu_{n, t}}{\sqrt{\sigma_{n, t}^2 + \epsilon}} \gamma_d + \beta_d.

🏭 Production Trade-offs

Why NLP uses LayerNorm: In NLP, sequences have variable lengths and batch sizes can vary dynamically down to 1 during inference. BatchNorm over sequences causes zero-padding tokens to corrupt channel statistics and introduces train-test discrepancies. LayerNorm operates completely independently of other samples in the batch.
⚠️ Common Interview Pitfalls
  • ✕
    Using BatchNorm on small batch sizes (B≤4B \le 4), leading to noisy variance estimates and catastrophic training degradation
  • ✕
    Forgetting to call model.eval() before running inference with BatchNorm, causing predictions to vary depending on test batch composition
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does BatchNorm fail on small batch sizes, and why doesn't LayerNorm suffer from this problem?
  • ?
    How does SyncBatchNorm coordinate cross-GPU statistics in distributed multi-GPU training?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-019: GELU vs SiLU: Differences, Connections, and Industry Model Adoption📋 Back to BankNext →M3-021: RMSNorm vs LayerNorm: Mathematical Differences and Why LLaMA Adopted RMSNorm