Batch Normalization

Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input $x$, compute the batch mean and variance $\mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i$, $\sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2$, normalize $\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$, then apply a learned affine transform $y = \gamma\hat{x} + \beta$ (per-channel $\gamma, \beta$ for CNNs; $\epsilon$ around 1e-5 prevents division by zero).

the default for image CNNs (ResNet and successors) to stabilize training, allow larger learning rates, and reduce sensitivity to initialization and dropout; interviews focus on the train/inference statistics gap, why small batches break it, and why NLP/Transformers switched to LayerNorm.

keeps activations at zero mean / unit variance, smoothing the loss landscape so convergence is often several times faster (training epochs can be roughly halved), with a mild regularizing side effect from batch-statistics noise. Key engineering point: training uses current batch statistics, while inference uses exponentially moving averages (EMA, momentum usually 0.1) accumulated during training — so small batches (batch=1 makes $\sigma_B=0$) distort the statistics and create train/inference mismatch, effectively breaking the model, an inherent weakness of BN.

Write the full BatchNorm computation: how $\mu_B$ and $\sigma_B^2$ are computed, the normalization $\hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}$, and the role of the affine parameters $\gamma,\beta$.

Train vs inference behavior: why must inference use the running mean/variance (EMA) instead of batch statistics, and when is the EMA updated?

Why does BN fail with small batches? What happens at batch=1, and what problems arise from train/inference statistic mismatch?

Why do Transformers/LLMs use LayerNorm instead of BatchNorm? What issues does BN face with variable-length sequences and padding?

Why does BN speed up convergence (ICS / loss-surface smoothing)? What is its regularizing side effect, and what order is used when combining it with Dropout?