🧠 Deep LearningID: batchnorm

Batch Normalization

Q: Write the full BatchNorm computation: how $\mu_B$ and $\sigma_B^2$ are computed, the normalization $\hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}$, and the role of the affine parameters $\gamma,\beta$.

【Batch Normalization 核心考点 1】Write the full BatchNorm computation: how $\mu_B$ and $\sigma_B^2$ are computed, the normalization $\hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}$, and the role of the affine parameters $\gamma,\beta$.。技术解析：Definition: Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input x, compute the batch mean and variance \mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2, nor... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/batchnorm

Q: Train vs inference behavior: why must inference use the running mean/variance (EMA) instead of batch statistics, and when is the EMA updated?

【Batch Normalization 核心考点 2】Train vs inference behavior: why must inference use the running mean/variance (EMA) instead of batch statistics, and when is the EMA updated?。技术解析：Definition: Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input x, compute the batch mean and variance \mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2, nor... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/batchnorm

Q: Why does BN fail with small batches? What happens at batch=1, and what problems arise from train/inference statistic mismatch?

【Batch Normalization 核心考点 3】Why does BN fail with small batches? What happens at batch=1, and what problems arise from train/inference statistic mismatch?。技术解析：Definition: Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input x, compute the batch mean and variance \mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2, nor... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/batchnorm

Q: Why do Transformers/LLMs use LayerNorm instead of BatchNorm? What issues does BN face with variable-length sequences and padding?

【Batch Normalization 核心考点 4】Why do Transformers/LLMs use LayerNorm instead of BatchNorm? What issues does BN face with variable-length sequences and padding?。技术解析：Definition: Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input x, compute the batch mean and variance \mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2, nor... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/batchnorm

Q: Why does BN speed up convergence (ICS / loss-surface smoothing)? What is its regularizing side effect, and what order is used when combining it with Dropout?

【Batch Normalization 核心考点 5】Why does BN speed up convergence (ICS / loss-surface smoothing)? What is its regularizing side effect, and what order is used when combining it with Dropout?。技术解析：Definition: Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input x, compute the batch mean and variance \mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i, \sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2, nor... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/batchnorm

BatchNorm 批归一化

🎯Core Definition

Batch Normalization normalizes activations along the batch axis per feature channel to combat internal covariate shift. Given input

x

, compute the batch mean and variance

\mu_B = \frac{1}{m}\sum_{i=1}^{m} x_i

\sigma_B^2 = \frac{1}{m}\sum_{i=1}^{m}(x_i - \mu_B)^2

, normalize

\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

, then apply a learned affine transform

y = \gamma\hat{x} + \beta

(per-channel

\gamma, \beta

for CNNs;

\epsilon

around 1e-5 prevents division by zero).

💡Use Cases

the default for image CNNs (ResNet and successors) to stabilize training, allow larger learning rates, and reduce sensitivity to initialization and dropout; interviews focus on the train/inference statistics gap, why small batches break it, and why NLP/Transformers switched to LayerNorm.

⚡Key Problems Solved

keeps activations at zero mean / unit variance, smoothing the loss landscape so convergence is often several times faster (training epochs can be roughly halved), with a mild regularizing side effect from batch-statistics noise. Key engineering point: training uses current batch statistics, while inference uses exponentially moving averages (EMA, momentum usually 0.1) accumulated during training — so small batches (batch=1 makes

\sigma_B=0

) distort the statistics and create train/inference mismatch, effectively breaking the model, an inherent weakness of BN.

🎯5 High-Frequency Exam Points

Write the full BatchNorm computation: how

\mu_B

and

\sigma_B^2

are computed, the normalization

\hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}

, and the role of the affine parameters

\gamma,\beta

Train vs inference behavior: why must inference use the running mean/variance (EMA) instead of batch statistics, and when is the EMA updated?

Why does BN fail with small batches? What happens at batch=1, and what problems arise from train/inference statistic mismatch?

Why do Transformers/LLMs use LayerNorm instead of BatchNorm? What issues does BN face with variable-length sequences and padding?

Why does BN speed up convergence (ICS / loss-surface smoothing)? What is its regularizing side effect, and what order is used when combining it with Dropout?

📖 In-depth Guide:📄 normalization-and-regularization →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Batch Normalization"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardLoss Functions Next Card →LayerNorm & RMSNorm

🔗 More Deep Learning Knowledge Cards

Activation Functions Adam & AdamW Autograd Compute Graph Classic CNN Architectures