B4MediumPart B · Normalization Family

Group Normalization & Instance Normalization

⚡Flashcard Drill➔

Partition channels into G groups and normalize independently per sample, removing batch-size dependence.

⏱️ Time: O(B * C * H * W)
💾 Space: O(1) 原地归一化
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def group_norm(
x: np.ndarray,
gamma: np.ndarray,
beta: np.ndarray,
num_groups: int = 4,
eps: float = 1e-5
) -> np.ndarray:
"""
GroupNorm 工业纯手写实现。
参数:
x: (B, C, H, W)
gamma, beta: (1, C, 1, 1) 可学习参数
num_groups: 分组数 G,要求 C % G == 0
"""
B, C, H, W = x.shape
assert C % num_groups == 0, f"通道数 {C} 必须能被组数 {num_groups} 整除"
channels_per_group =
# 核心步骤:重塑为 (B, G, C//G, H, W)
x_reshaped = x.reshape(B, num_groups, channels_per_group, H, W)
# 沿组内轴 (2, 3, 4) 求均值与方差
mean = np.mean(x_reshaped, axis=(2, 3, 4), keepdims=True)
var = np.var(x_reshaped, axis=(2, 3, 4), keepdims=True)
# 组内归一化
x_norm = (x_reshaped - mean) / np.sqrt(var + eps)
# 还原形状回 (B, C, H, W)
x_norm = x_norm.reshape(B, C, H, W)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why do diffusion U-Nets commonly prefer GroupNorm over LayerNorm?
Next: Inverted Dropout with Train/Eval Scaling →
💡

Core Mental Anchor / Mnemonic

Reshape channels into G groups, normalize across (C/G, H, W) per sample regardless of batch size.

📐 Mathematical Derivation & Core Formula

x∈RB×C×H×W→Reshape(B,G,C/G,H,W)  ⟹  μ,σ over (C/G,H,W)x \in \mathbb{R}^{B \times C \times H \times W} \xrightarrow{\text{Reshape}} (B, G, C/G, H, W) \implies \mu, \sigma \text{ over } (C/G, H, W)
### Channel Grouping & Special Cases
When batch sizes are small (e.g., 1 or 2 in object detection, segmentation, or high-resolution image synthesis), BatchNorm batch statistics become noisy and destabilize training.
GroupNorm (Wu & He, 2018):
- Divides CC channels into GG groups of C/GC/G channels each;
- Computes mean and variance over (C/G,H,W)(C/G, H, W) within each sample;
- Special case $G = 1$: Reduces to LayerNorm over all channels;
- Special case $G = C$: Reduces to InstanceNorm per channel (common in style transfer).

🔄 Tensor Dimensions & Shape Flow

(B, C, H, W) -> reshape (B, G, C//G, H, W) -> mean/var on (2,3,4) -> normalize -> reshape (B, C, H, W) -> affine

🛡️ Numerical Stability & Pitfalls Checklist

  • Ensure channel count C is evenly divisible by num_groups G.
  • Diffusion U-Nets commonly replace BatchNorm with GroupNorm(32) for small-batch stability.
  • Behavior is identical in training and evaluation modes.

💻 Reference Code Implementation

import numpy as np

def group_norm(
    x: np.ndarray,
    gamma: np.ndarray,
    beta: np.ndarray,
    num_groups: int = 4,
    eps: float = 1e-5
) -> np.ndarray:
    """
    GroupNorm 工业纯手写实现。
    参数:
        x: (B, C, H, W)
        gamma, beta: (1, C, 1, 1) 可学习参数
        num_groups: 分组数 G,要求 C % G == 0
    """
    B, C, H, W = x.shape
    assert C % num_groups == 0, f"通道数 {C} 必须能被组数 {num_groups} 整除"
    channels_per_group = C // num_groups
    
    # 核心步骤:重塑为 (B, G, C//G, H, W)
    x_reshaped = x.reshape(B, num_groups, channels_per_group, H, W)
    
    # 沿组内轴 (2, 3, 4) 求均值与方差
    mean = np.mean(x_reshaped, axis=(2, 3, 4), keepdims=True)
    var = np.var(x_reshaped, axis=(2, 3, 4), keepdims=True)
    
    # 组内归一化
    x_norm = (x_reshaped - mean) / np.sqrt(var + eps)
    
    # 还原形状回 (B, C, H, W)
    x_norm = x_norm.reshape(B, C, H, W)
    
    return gamma * x_norm + beta

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.random.randn(2, 8, 4, 4)
gamma = np.ones((1, 8, 1, 1))
beta = np.zeros((1, 8, 1, 1))
out = group_norm(x, gamma, beta, num_groups=4)
assert out.shape == (2, 8, 4, 4)
# 验证每个样本内每个组的均值为 0
out_g = out.reshape(2, 4, 2, 4, 4)
assert np.allclose(out_g.mean(axis=(2, 3, 4)), 0.0, atol=1e-3)
print("✓ GroupNorm 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why do diffusion U-Nets commonly prefer GroupNorm over LayerNorm?
Image feature maps exhibit strong spatial locality and cross-channel correlation within semantic groups. GroupNorm lets channels within each group share normalization statistics while keeping channel groups independent, which empirically works well for spatial visual features.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: BatchNorm1d with Train vs Eval DistinctionBack to Part B & All 105Next: Inverted Dropout with Train/Eval Scaling →
中文速览与面试要点

GroupNorm 与 InstanceNorm(分组与实例归一化)

何恺明团队经典架构,将通道划分为 G 个组独立归一化,彻底解除对 Batch Size 大小的依赖。

核心要点

  • ### 通道解耦与几何物理意义 在小 Batch Size(如检测、分割、多模态高分辨率图像生成任务中 Batch=1 或 2)下,BatchNorm 统计量剧烈抖动导致训练发散。 何恺明等人在 2018 年提出 GroupNorm(GN): - 将 C 个通道划分为 G 个组(Group),每组包含 C/G 个通道; - 沿每组的 (C/G, H, W) 计算均值和方差; - 极端特例 1:当 G = 1 时,退化为针对整张图的 LayerNorm; - 极端特例 2:当 G = C 时,退化为针对单个通道的 InstanceNorm(常用于风格迁移和音频合成)。
  • (B, C, H, W) -> reshape -> (B, G, C//G, H, W) -> mean/var on axis=(2,3,4) -> (B, G, 1, 1, 1) -> 归一化 -> 还原 (B, C, H, W) -> gamma/beta 仿射
  • 必须确保 C % num_groups == 0
  • 在 Diffusion 架构(如 Stable Diffusion / DiT)中,ResNetBlock 普遍采用 GroupNorm(32) 代替 BatchNorm
  • 训练和测试完全无需切换模式,表现恒定一致
  • 为什么在扩散模型(Diffusion U-Net / DiT)中倾向于使用 GroupNorm 而非 LayerNorm?:图像特征具有强烈的空间局部关联性与多通道相关性。GroupNorm 允许同一语义分组内的不同特征通道(如边缘与纹理)共享统计量,在保持通道表达独立性的同时平滑空间激活,实验证明在生成画质上显著优于 LayerNorm。

参考实现

import numpy as np

def group_norm(
    x: np.ndarray,
    gamma: np.ndarray,
    beta: np.ndarray,
    num_groups: int = 4,
    eps: float = 1e-5
) -> np.ndarray:
    """
    GroupNorm 工业纯手写实现。
    参数:
        x: (B, C, H, W)
        gamma, beta: (1, C, 1, 1) 可学习参数
        num_groups: 分组数 G,要求 C % G == 0
    """
    B, C, H, W = x.shape
    assert C % num_groups == 0, f"通道数 {C} 必须能被组数 {num_groups} 整除"
    channels_per_group = C // num_groups
    
    # 核心步骤:重塑为 (B, G, C//G, H, W)
    x_reshaped = x.reshape(B, num_groups, channels_per_group, H, W)
    
    # 沿组内轴 (2, 3, 4) 求均值与方差
    mean = np.mean(x_reshaped, axis=(2, 3, 4), keepdims=True)
    var = np.var(x_reshaped, axis=(2, 3, 4), keepdims=True)
    
    # 组内归一化
    x_norm = (x_reshaped - mean) / np.sqrt(var + eps)
    
    # 还原形状回 (B, C, H, W)
    x_norm = x_norm.reshape(B, C, H, W)
    
    return gamma * x_norm + beta