← Back to Deep Learning Mind Map
🧠 Deep LearningID: normalization-comparison

Norm Methods Comparison

BN vs LN vs GN 对比
🎯Core Definition
the three normalization families differ only in which axis their statistics are computed over (input x∈RN×C×H×Wx \in \mathbb{R}^{N \times C \times H \times W}): • BN: statistics over (N, H, W) — across batch and spatial dims, per channel: x^c=xc−μB,cσB,c2+ϵ\hat{x}_c = \frac{x_c - \mu_{B,c}}{\sqrt{\sigma_{B,c}^2 + \epsilon}}; • LN: statistics over (C, H, W) — within one sample across all channels: μ=1CHW∑x\mu = \frac{1}{CHW}\sum x; • GN: statistics over (C/G, H, W) — within one sample, channels are split into GG groups (e.g. 32) normalized independently.
💡Use Cases
BN for large-batch CNN classification; LN for Transformers/LLMs/RNNs (batch-independent, friendly to variable-length sequences); GN for small-batch tasks (detection/segmentation at 1–2 images per GPU) and 3D/video where BN fails. Interviews usually ask for a per-method table of statistic axes plus each one's failure mode.
⚡Key Problems Solved
normalization stabilizes activation distributions for faster convergence and larger learning rates; the real trade-off is 'how many samples/channels share a statistic': BN depends on batch statistics (distorted at small batch, train/inference mismatch), LN needs no batch at all but assumes channels are i.i.d. (less apt for CNN feature maps, weaker regularization), and GN is the compromise — per-sample group statistics stay stable and batch-size-independent, making it the standard for detection at 1–2 images per GPU.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of Norm Methods Comparison.
Use Cases
In production engineering, what are the primary use cases for Norm Methods Comparison and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
For x∈RN×C×H×Wx \in \mathbb{R}^{N \times C \times H \times W}, compare the normalization axes of BN/LN/GN: what is the statistic range of each (batch/sample/channel)?
2
Why is BN unsuitable for Transformers/variable-length sequences while LN is the default? Compare their batch dependency.
3
What problem does GroupNorm solve? Why does BN fail on small-batch detection/segmentation (1–2 images/GPU) while GN works?
4
What are BN's small-batch weakness and LN's channel-independence assumption? In which settings is LN worse than BN?
5
Why does normalization speed up training? Contrast the internal-covariate-shift view with the loss-landscape-smoothing view.
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "Norm Methods Comparison"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardLayerNorm & RMSNormNext Card →Dropout

🔗 More Deep Learning Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/DL/card/normalization-comparison
中文速览与面试要点

BN vs LN vs GN 对比 (Norm Methods Comparison)

定义: 三种主流归一化的本质差异在统计量的计算维度 (输入 x R^N C H W): • BN (BatchNorm): 统计范围 (N, H, W) — 跨 batch 与空间维, 每通道独立, x_c = x_c - _B,c _B,c^2 + ; • LN (LayerNorm): 统计范围 (C, H, W) — 单样本内跨全部通道, = 1CHW x; • GN (GroupNorm): 统计范围 (C/G, H, W) — 单样本内把通道分成 G 组 (如 32), 组内归一化。 使用场景: BN — 大 batch 的 CNN 分类; LN — Transformer/LLM/RNN (与 batch 无关, 变长序列友好); GN — 小 batch 任务 (检测/分割, 每卡 1~2 张图) 及 BN 无法工作的 3D/视频。面试常考: 一张表对比三者统计维度 + 各自失效场景。 解决的问题: 归一化稳定激活分布、加速收敛并允许更大学习率; 取舍本质是“统计量跨多少样本/通道共享”: BN 依赖 batch 统计 (小 batch 失真, 训练/推理不一致), LN 完全不依赖 batch 但假设各通道同分布 (对 CNN 特征图不如 BN 贴切, 正则化也更弱), GN 是折中——引入通道分组统计, 小 batch 下稳定且与 batch size 无关, 已成为每卡 1~2 张图的检测任务标配。

核心要点

  • 对 x R^N C H W, 画出 BN/LN/GN 的归一化维度对比: 各自统计量的计算范围 (batch/样本/通道) 是什么?
  • 为什么 BN 在 Transformer/变长序列中不适用而 LN 成为标配? 两者的 batch 依赖差异?
  • GN 解决什么问题? 在小 batch (每卡 1~2 张) 的检测/分割任务中为什么 BN 失效而 GN 可行?
  • BN 的小 batch 缺陷与 LN 的通道独立性假设各是什么? 什么场景下 LN 不如 BN?
  • 归一化为什么能加速训练? 对比内部协变量偏移 (ICS) 与 loss 曲面平滑两种解释视角?