B3Medium★ Core Essential · Industrial BedrockPart B · Normalization Family

BatchNorm1d with Train vs Eval Distinction

⚡Flashcard Drill➔

Track batch statistics with momentum into running stats during training; freeze and apply running stats at inference.

⏱️ Time: O(B * D) 线性操作
💾 Space: O(D) 维护 running_mean 与 running_var 两个全局向量
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class BatchNorm1d:
def __init__(self, num_features: int, eps: float = 1e-5, momentum: float = 0.1):
self.num_features = num_features
self.eps = eps
self.momentum = momentum
# 可学习参数
self.gamma = np.ones(num_features)
self.beta = np.zeros(num_features)
# 运行统计量 (推理期使用)
self.running_mean = np.zeros(num_features)
self.running_var = np.ones(num_features)
def forward(self, x: np.ndarray, training: bool = True) -> np.ndarray:
"""
x 形状: (B, D)
"""
if training:
# 沿 Batch 轴 (axis=0) 计算批次统计量
batch_mean =
batch_var = np.var(x, axis=0)
# 归一化
x_hat = (x - batch_mean) / np.sqrt(batch_var + self.eps)
# 更新全局动量统计量 (PyTorch 默认无偏校正样本方差)
m = self.momentum
self.running_mean = (1 - m) * self.running_mean + m * batch_mean
self.running_var = (1 - m) * self.running_var + m * batch_var
else:
# 推理阶段必须使用 running stats
x_hat = (x - self.running_mean) / np.sqrt(self.running_var + self.eps)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is BatchNorm rarely used in NLP or Transformer architectures?
Next: Group Normalization & Instance Normalization →
💡

Core Mental Anchor / Mnemonic

Train on mini-batch stats with running EMA; freeze to running stats during inference.

📐 Mathematical Derivation & Core Formula

Train: x^=x−μBσB2+ϵ,Eval: x^=x−μrunσrun2+ϵ\text{Train: } \hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad \text{Eval: } \hat{x} = \frac{x - \mu_{\text{run}}}{\sqrt{\sigma_{\text{run}}^2 + \epsilon}}
### Core Train vs. Eval Behavioral Difference
BatchNorm normalizes each feature channel across the batch (and spatial dimensions):
1. Training (`training=True`):
- Normalizes using the current mini-batch mean μB\mu_B and variance σB2\sigma_B^2;
- Updates global running statistics via exponential moving average:
μrun←(1−m)μrun+m⋅μB\mu_{\text{run}} \leftarrow (1 - m) \mu_{\text{run}} + m \cdot \mu_B
σrun2←(1−m)σrun2+m⋅σB,unbiased2\sigma_{\text{run}}^2 \leftarrow (1 - m) \sigma_{\text{run}}^2 + m \cdot \sigma_{B,\text{unbiased}}^2
2. Evaluation (`training=False`):
- Freezes statistics and uses stored μrun\mu_{\text{run}} and σrun2\sigma_{\text{run}}^2 so predictions for a single sample are deterministic and independent of batch peers.

🔄 Tensor Dimensions & Shape Flow

Train: (B, D) -> batch_mean/var: (D,) -> normalize & update running -> affine; Eval: (B, D) -> frozen running_mean/var: (D,)

🛡️ Numerical Stability & Pitfalls Checklist

  • Never use current batch stats in eval mode (batch_size=1 collapses variance to zero).
  • Frameworks like PyTorch apply Bessel's correction N/(N-1) when updating running_var.
  • Fold BatchNorm affine parameters into preceding Conv/Linear weights before deployment (Conv-BN fusion).

💻 Reference Code Implementation

import numpy as np

class BatchNorm1d:
    def __init__(self, num_features: int, eps: float = 1e-5, momentum: float = 0.1):
        self.num_features = num_features
        self.eps = eps
        self.momentum = momentum
        
        # 可学习参数
        self.gamma = np.ones(num_features)
        self.beta = np.zeros(num_features)
        
        # 运行统计量 (推理期使用)
        self.running_mean = np.zeros(num_features)
        self.running_var = np.ones(num_features)
        
    def forward(self, x: np.ndarray, training: bool = True) -> np.ndarray:
        """
        x 形状: (B, D)
        """
        if training:
            # 沿 Batch 轴 (axis=0) 计算批次统计量
            batch_mean = np.mean(x, axis=0)
            batch_var = np.var(x, axis=0)
            
            # 归一化
            x_hat = (x - batch_mean) / np.sqrt(batch_var + self.eps)
            
            # 更新全局动量统计量 (PyTorch 默认无偏校正样本方差)
            m = self.momentum
            self.running_mean = (1 - m) * self.running_mean + m * batch_mean
            self.running_var = (1 - m) * self.running_var + m * batch_var
        else:
            # 推理阶段必须使用 running stats
            x_hat = (x - self.running_mean) / np.sqrt(self.running_var + self.eps)
            
        return self.gamma * x_hat + self.beta

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
bn = BatchNorm1d(num_features=4)
x_train = np.random.randn(32, 4) * 5 + 3
out_tr = bn.forward(x_train, training=True)
assert np.allclose(out_tr.mean(axis=0), 0.0, atol=1e-3)
x_test = np.random.randn(1, 4)
out_te = bn.forward(x_test, training=False)
assert out_te.shape == (1, 4), "推理输出形状有误"
print("✓ BatchNorm1d 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why is BatchNorm rarely used in NLP or Transformer architectures?
Text sequences vary widely in length (introducing padded positions), causing high statistical variance across positions within a batch. More importantly, autoregressive inference feeds one token step at a time (effective batch size 1 along token/context), where batch statistics are unreliable.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Root Mean Square Normalization (RMSNorm)Back to Part B & All 105Next: Group Normalization & Instance Normalization →