B1Easy★ Core Essential · Industrial BedrockPart B · Normalization Family
Layer Normalization with Affine Transformation
Per-token normalization across the feature dimension with consistent train/eval behavior and learned gamma/beta.
⏱️ Time:
O(B * S * D) 线性耗时💾 Space:
O(1) 无额外持久化状态,推理零额外显存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def layer_norm(
x: np.ndarray,
gamma: np.ndarray,
beta: np.ndarray,
eps: float = 1e-5
) -> np.ndarray:
"""
纯手写 LayerNorm 算子。
参数:
x: (B, S, D) 输入激活
gamma: (D,) 可学习缩放仿射系数
beta: (D,) 可学习偏置仿射系数
eps: 防止方差为 0 的数值缓冲
返回:
out: (B, S, D)
"""
# 沿特征轴 (axis=-1) 计算均值与方差,保留维度
mean =
var = np.var(x, axis=-1, keepdims=True)
# 归一化
x_hat = (x - mean) / np.sqrt(var + eps)
# 仿射变换
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What are the key differences and trade-offs between Pre-LN and Post-LN Transformer blocks?
💡
Core Mental Anchor / Mnemonic
Normalize along the last axis inside sqrt(var + eps), keep dims, then scale with gamma and shift with beta.
📐 Mathematical Derivation & Core Formula
### Mathematical Formulation & Dimension Mechanics
For input , LayerNorm computes mean and variance independently along the final feature dimension :
Unlike BatchNorm, which aggregates across samples in a batch, LayerNorm computes statistics per token within each sequence:
1. Batch-size independence: Works naturally for variable sequence lengths and batch size 1 during autoregressive decoding;
2. Train/eval consistency: No running statistics (
For input , LayerNorm computes mean and variance independently along the final feature dimension :
Unlike BatchNorm, which aggregates across samples in a batch, LayerNorm computes statistics per token within each sequence:
1. Batch-size independence: Works naturally for variable sequence lengths and batch size 1 during autoregressive decoding;
2. Train/eval consistency: No running statistics (
running_mean or running_var) need to be tracked.🔄 Tensor Dimensions & Shape Flow
(B, S, D) -> mean: (B, S, 1) -> var: (B, S, 1) -> x_hat: (B, S, D) -> gamma * x_hat + beta -> (B, S, D)
🛡️ Numerical Stability & Pitfalls Checklist
- Add eps (typically 1e-5) inside the square root in the denominator, never outside.
- Keep keepdims=True so (B, S, 1) broadcasts cleanly against (B, S, D).
- Sum gradients for gamma and beta across the (B, S) axes during backpropagation.
💻 Reference Code Implementation
import numpy as np
def layer_norm(
x: np.ndarray,
gamma: np.ndarray,
beta: np.ndarray,
eps: float = 1e-5
) -> np.ndarray:
"""
纯手写 LayerNorm 算子。
参数:
x: (B, S, D) 输入激活
gamma: (D,) 可学习缩放仿射系数
beta: (D,) 可学习偏置仿射系数
eps: 防止方差为 0 的数值缓冲
返回:
out: (B, S, D)
"""
# 沿特征轴 (axis=-1) 计算均值与方差,保留维度
mean = np.mean(x, axis=-1, keepdims=True)
var = np.var(x, axis=-1, keepdims=True)
# 归一化
x_hat = (x - mean) / np.sqrt(var + eps)
# 仿射变换
return gamma * x_hat + beta
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
x = np.random.randn(2, 4, 8)
gamma = np.ones(8)
beta = np.zeros(8)
out = layer_norm(x, gamma, beta)
# 验证最后一维的均值为 0,方差为 1
assert np.allclose(out.mean(axis=-1), 0.0, atol=1e-6)
assert np.allclose(out.var(axis=-1), 1.0, atol=1e-3)
print("✓ LayerNorm 自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What are the key differences and trade-offs between Pre-LN and Post-LN Transformer blocks?
Post-LN applies LayerNorm after the residual addition, which weakens gradient propagation in deep stacks and makes training sensitive to learning-rate warmup. Pre-LN applies LayerNorm before the attention and FFN sub-layers so the residual path remains an identity highway, improving training stability in deep large language models.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.