B1Easy★ Core Essential · Industrial BedrockPart B · Normalization Family

Layer Normalization with Affine Transformation

⚡Flashcard Drill➔

Per-token normalization across the feature dimension with consistent train/eval behavior and learned gamma/beta.

⏱️ Time: O(B * S * D) 线性耗时
💾 Space: O(1) 无额外持久化状态,推理零额外显存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def layer_norm(
x: np.ndarray,
gamma: np.ndarray,
beta: np.ndarray,
eps: float = 1e-5
) -> np.ndarray:
"""
纯手写 LayerNorm 算子。
参数:
x: (B, S, D) 输入激活
gamma: (D,) 可学习缩放仿射系数
beta: (D,) 可学习偏置仿射系数
eps: 防止方差为 0 的数值缓冲
返回:
out: (B, S, D)
"""
# 沿特征轴 (axis=-1) 计算均值与方差,保留维度
mean =
var = np.var(x, axis=-1, keepdims=True)
# 归一化
x_hat = (x - mean) / np.sqrt(var + eps)
# 仿射变换
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What are the key differences and trade-offs between Pre-LN and Post-LN Transformer blocks?
Next: Root Mean Square Normalization (RMSNorm) →
💡

Core Mental Anchor / Mnemonic

Normalize along the last axis inside sqrt(var + eps), keep dims, then scale with gamma and shift with beta.

📐 Mathematical Derivation & Core Formula

x^=x−μσ2+ϵ,y=γ⊙x^+β\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}, \quad y = \gamma \odot \hat{x} + \beta
### Mathematical Formulation & Dimension Mechanics
For input x∈RB×S×Dx \in \mathbb{R}^{B \times S \times D}, LayerNorm computes mean and variance independently along the final feature dimension DD:
μ=1D∑i=1Dxi,σ2=1D∑i=1D(xi−μ)2\mu = \frac{1}{D} \sum_{i=1}^D x_i, \quad \sigma^2 = \frac{1}{D} \sum_{i=1}^D (x_i - \mu)^2
Unlike BatchNorm, which aggregates across samples in a batch, LayerNorm computes statistics per token within each sequence:
1. Batch-size independence: Works naturally for variable sequence lengths and batch size 1 during autoregressive decoding;
2. Train/eval consistency: No running statistics (running_mean or running_var) need to be tracked.

🔄 Tensor Dimensions & Shape Flow

(B, S, D) -> mean: (B, S, 1) -> var: (B, S, 1) -> x_hat: (B, S, D) -> gamma * x_hat + beta -> (B, S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Add eps (typically 1e-5) inside the square root in the denominator, never outside.
  • Keep keepdims=True so (B, S, 1) broadcasts cleanly against (B, S, D).
  • Sum gradients for gamma and beta across the (B, S) axes during backpropagation.

💻 Reference Code Implementation

import numpy as np

def layer_norm(
    x: np.ndarray,
    gamma: np.ndarray,
    beta: np.ndarray,
    eps: float = 1e-5
) -> np.ndarray:
    """
    纯手写 LayerNorm 算子。
    参数:
        x: (B, S, D) 输入激活
        gamma: (D,) 可学习缩放仿射系数
        beta:  (D,) 可学习偏置仿射系数
        eps:   防止方差为 0 的数值缓冲
    返回:
        out: (B, S, D)
    """
    # 沿特征轴 (axis=-1) 计算均值与方差,保留维度
    mean = np.mean(x, axis=-1, keepdims=True)
    var = np.var(x, axis=-1, keepdims=True)
    
    # 归一化
    x_hat = (x - mean) / np.sqrt(var + eps)
    
    # 仿射变换
    return gamma * x_hat + beta

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.random.randn(2, 4, 8)
gamma = np.ones(8)
beta = np.zeros(8)
out = layer_norm(x, gamma, beta)
# 验证最后一维的均值为 0,方差为 1
assert np.allclose(out.mean(axis=-1), 0.0, atol=1e-6)
assert np.allclose(out.var(axis=-1), 1.0, atol=1e-3)
print("✓ LayerNorm 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What are the key differences and trade-offs between Pre-LN and Post-LN Transformer blocks?
Post-LN applies LayerNorm after the residual addition, which weakens gradient propagation in deep stacks and makes training sensitive to learning-rate warmup. Pre-LN applies LayerNorm before the attention and FFN sub-layers so the residual path remains an identity highway, improving training stability in deep large language models.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Softplus & LeakyReLU with Overflow GuardsBack to Part B & All 105Next: Root Mean Square Normalization (RMSNorm) →