Pre-LN & RMSNorm

LayerNorm standardizes the last dimension: $LN(x) = \frac{x - mean}{\sqrt{var}} \cdot \gamma + \beta$; Pre-LN places normalization before the residual sublayer ($x + SubLayer(LN(x))$) while Post-LN applies it after the residual addition; RMSNorm drops the mean and scales by root-mean-square only: $RMSNorm(x) = \frac{x}{\sqrt{mean(x^2)}} \cdot \gamma$.

mainstream models (LLaMA, Mistral, Qwen) uniformly use Pre-LN + RMSNorm; classic interview questions are "why Pre-LN trains deep models stably" and "why RMSNorm can drop the mean".

in Post-LN the residual signal is progressively attenuated by normalization, making deep training unstable and requiring delicate warmup and gradient clipping; Pre-LN lets gradients flow back through the near-identity residual path, stabilizing depth; RMSNorm removes the mean computation (saving about 1/3 of normalization FLOPs) and experiments show re-centering is essentially unnecessary for Transformers.

Structural difference between Pre-LN and Post-LN? Why is Pre-LN more stable for deep models?

RMSNorm formula? Difference from LayerNorm, and why dropping the mean works?

Downsides of Pre-LN (residual drift; late-training accuracy vs Post-LN, warmup switching)?

Why does the position of normalization relative to the residual matter? Why must the $\gamma$ scale be kept?

What problem does DeepNorm solve (stable training of 1000-layer Post-LN models)?