Norm Methods Comparison

the three normalization families differ only in which axis their statistics are computed over (input $x \in \mathbb{R}^{N \times C \times H \times W}$): • BN: statistics over (N, H, W) — across batch and spatial dims, per channel: $\hat{x}_c = \frac{x_c - \mu_{B,c}}{\sqrt{\sigma_{B,c}^2 + \epsilon}}$; • LN: statistics over (C, H, W) — within one sample across all channels: $\mu = \frac{1}{CHW}\sum x$; • GN: statistics over (C/G, H, W) — within one sample, channels are split into $G$ groups (e.g. 32) normalized independently.

BN for large-batch CNN classification; LN for Transformers/LLMs/RNNs (batch-independent, friendly to variable-length sequences); GN for small-batch tasks (detection/segmentation at 1–2 images per GPU) and 3D/video where BN fails. Interviews usually ask for a per-method table of statistic axes plus each one's failure mode.

normalization stabilizes activation distributions for faster convergence and larger learning rates; the real trade-off is 'how many samples/channels share a statistic': BN depends on batch statistics (distorted at small batch, train/inference mismatch), LN needs no batch at all but assumes channels are i.i.d. (less apt for CNN feature maps, weaker regularization), and GN is the compromise — per-sample group statistics stay stable and batch-size-independent, making it the standard for detection at 1–2 images per GPU.

Test Your Knowledge: Practice Questions for "Norm Methods Comparison"

Single choice pitfall questions with instant feedback and mistake tracking.