LayerNorm & RMSNorm

LayerNorm normalizes a single sample across its feature dimension, independent of the batch: for $x \in \mathbb{R}^d$, compute $\mu = \frac{1}{d}\sum_{i=1}^{d}x_i$, $\sigma^2 = \frac{1}{d}\sum_{i=1}^{d}(x_i-\mu)^2$, giving $y = \gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} + \beta$. RMSNorm is a simplification that drops the centering: $y = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d}x_i^2 + \epsilon}}\cdot\gamma$.

the default for Transformers / NLP / RNNs (variable-length sequences make batch statistics unreliable); RMSNorm is the standard choice for LLMs such as LLaMA and Qwen (dropping mean and bias computation, roughly 7%–30% faster training). Interviews ask why LN is batch-independent, why RMSNorm is sufficient, and Pre-Norm placement.

removes the dependency on the batch — works at batch=1 and avoids padding pollution on variable-length sequences. Why omitting centering still works: recentering/rescaling invariance is inherent per-layer, and the mean term is absorbed by subsequent residual layers; the cost is weaker regularization than BN (no cross-sample statistical noise).

Write the LayerNorm formula $y = \gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta$, state which axis it normalizes over, and why it is independent of batch size.

How does RMSNorm differ from LayerNorm, why does removing the mean matter little, and what does it save in LLMs (compute / reported speedup)?

The role of $\gamma$ and $\beta$ in LN: what happens if they are removed, and why are they needed to restore representational power?

Pre-Norm vs Post-Norm: why do modern Transformers use Pre-LN (training stability and the residual gradient path)?

Why must RNNs/variable-length sequences use LayerNorm rather than BatchNorm? Compare their batch dependency and padding handling.