Pre-LN & RMSNorm

LayerNorm standardizes the last dimension: $LN(x) = \frac{x - mean}{\sqrt{var}} \cdot \gamma + \beta$; Pre-LN places normalization before the residual sublayer ($x + SubLayer(LN(x))$) while Post-LN applies it after the residual addition; RMSNorm drops the mean and scales by root-mean-square only: $RMSNorm(x) = \frac{x}{\sqrt{mean(x^2)}} \cdot \gamma$.

mainstream models (LLaMA, Mistral, Qwen) uniformly use Pre-LN + RMSNorm; classic interview questions are "why Pre-LN trains deep models stably" and "why RMSNorm can drop the mean".

in Post-LN the residual signal is progressively attenuated by normalization, making deep training unstable and requiring delicate warmup and gradient clipping; Pre-LN lets gradients flow back through the near-identity residual path, stabilizing depth; RMSNorm removes the mean computation (saving about 1/3 of normalization FLOPs) and experiments show re-centering is essentially unnecessary for Transformers.

Test Your Knowledge: Practice Questions for "Pre-LN & RMSNorm"

Single choice pitfall questions with instant feedback and mistake tracking.