Norm Methods Comparison

the three normalization families differ only in which axis their statistics are computed over (input $x \in \mathbb{R}^{N \times C \times H \times W}$): • BN: statistics over (N, H, W) — across batch and spatial dims, per channel: $\hat{x}_c = \frac{x_c - \mu_{B,c}}{\sqrt{\sigma_{B,c}^2 + \epsilon}}$; • LN: statistics over (C, H, W) — within one sample across all channels: $\mu = \frac{1}{CHW}\sum x$; • GN: statistics over (C/G, H, W) — within one sample, channels are split into $G$ groups (e.g. 32) normalized independently.

BN for large-batch CNN classification; LN for Transformers/LLMs/RNNs (batch-independent, friendly to variable-length sequences); GN for small-batch tasks (detection/segmentation at 1–2 images per GPU) and 3D/video where BN fails. Interviews usually ask for a per-method table of statistic axes plus each one's failure mode.

normalization stabilizes activation distributions for faster convergence and larger learning rates; the real trade-off is 'how many samples/channels share a statistic': BN depends on batch statistics (distorted at small batch, train/inference mismatch), LN needs no batch at all but assumes channels are i.i.d. (less apt for CNN feature maps, weaker regularization), and GN is the compromise — per-sample group statistics stay stable and batch-size-independent, making it the standard for detection at 1–2 images per GPU.

For $x \in \mathbb{R}^{N \times C \times H \times W}$, compare the normalization axes of BN/LN/GN: what is the statistic range of each (batch/sample/channel)?

Why is BN unsuitable for Transformers/variable-length sequences while LN is the default? Compare their batch dependency.

What problem does GroupNorm solve? Why does BN fail on small-batch detection/segmentation (1–2 images/GPU) while GN works?

What are BN's small-batch weakness and LN's channel-independence assumption? In which settings is LN worse than BN?

Why does normalization speed up training? Contrast the internal-covariate-shift view with the loss-landscape-smoothing view.