B2Easy★ Core Essential · Industrial BedrockPart B · Normalization Family

Root Mean Square Normalization (RMSNorm)

⚡Flashcard Drill➔

Omit mean centering and normalize solely by root-mean-square to save memory passes in modern LLMs.

⏱️ Time: O(B * S * D) 相比 LayerNorm 减少约 30% 指令周期
💾 Space: O(1) 仅需保存 RMS 标量供反向传播
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def rms_norm(x: np.ndarray, gamma: np.ndarray, eps: float = 1e-6) -> np.ndarray:
"""
纯手写 RMSNorm 算子。
参数:
x: (B, S, D) 输入激活
gamma: (D,) 可学习缩放权重
eps: 数值稳定常数(LLaMA-3 采用 1e-5,Gemma 采用 1e-6)
返回:
out: (B, S, D)
"""
# 计算均方根 (RMS)
rms =
# 归一化并仿射缩放
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does Gemma's RMSNorm formulation differ subtly from LLaMA's?
Next: BatchNorm1d with Train vs Eval Distinction →
💡

Core Mental Anchor / Mnemonic

Skip mean subtraction: square, average, sqrt(mean_sq + eps), divide, and multiply by gamma.

📐 Mathematical Derivation & Core Formula

RMS(x)=1D∑i=1Dxi2+ϵ,y=xRMS(x)⊙γ\mathrm{RMS}(x) = \sqrt{\frac{1}{D} \sum_{i=1}^D x_i^2 + \epsilon}, \quad y = \frac{x}{\mathrm{RMS}(x)} \odot \gamma
### Design Philosophy & Memory-Access Reduction
Zhang & Sennrich (2019) showed that LayerNorm's primary benefit comes from re-scaling invariance rather than mean centering.
RMSNorm omits mean computation and subtraction, normalizing strictly by the root mean square:
RMS(x)=1D∥x∥2+ϵ\mathrm{RMS}(x) = \sqrt{\frac{1}{D} \|x\|^2 + \epsilon}
Key benefits:
1. Fewer global memory passes: Eliminates the mean pass and subtraction pass, helpful for memory-bandwidth-bound kernels;
2. Fewer parameters: Drops the additive bias β\beta, keeping only the scale parameter γ\gamma;
3. Widely adopted in LLaMA, Mistral, Gemma, and DeepSeek architectures.

🔄 Tensor Dimensions & Shape Flow

(B, S, D) -> mean(x^2) on last axis -> (B, S, 1) -> sqrt(var + eps) -> broadcast divide * gamma -> (B, S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • LLaMA and DeepSeek typically use eps around 1e-5 or 1e-6; overly large eps distorts feature scale.
  • Upcast x^2 accumulation and sqrt to FP32 in mixed-precision kernels before casting back to FP16/BF16.

💻 Reference Code Implementation

import numpy as np

def rms_norm(x: np.ndarray, gamma: np.ndarray, eps: float = 1e-6) -> np.ndarray:
    """
    纯手写 RMSNorm 算子。
    参数:
        x: (B, S, D) 输入激活
        gamma: (D,) 可学习缩放权重
        eps: 数值稳定常数(LLaMA-3 采用 1e-5,Gemma 采用 1e-6)
    返回:
        out: (B, S, D)
    """
    # 计算均方根 (RMS)
    rms = np.sqrt(np.mean(x ** 2, axis=-1, keepdims=True) + eps)
    # 归一化并仿射缩放
    return (x / rms) * gamma

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.random.randn(2, 3, 16)
gamma = np.ones(16)
out = rms_norm(x, gamma)
# 均方根应严格接近 1
rms_calc = np.sqrt(np.mean(out ** 2, axis=-1))
assert np.allclose(rms_calc, 1.0, atol=1e-3)
print("✓ RMSNorm 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does Gemma's RMSNorm formulation differ subtly from LLaMA's?
Gemma parameterizes the affine scale with a unit offset: y=xRMS(x)⊙(1+γ)y = \frac{x}{\mathrm{RMS}(x)} \odot (1 + \gamma) with γ\gamma initialized to 0. Even with zero-initialized weights, the layer starts as exact identity normalization.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Layer Normalization with Affine TransformationBack to Part B & All 105Next: BatchNorm1d with Train vs Eval Distinction →