B4MediumPart B · Normalization Family

Group Normalization & Instance Normalization

⚡Flashcard Drill➔

Partition channels into G groups and normalize independently per sample, removing batch-size dependence.

⏱️ Time: O(B * C * H * W)
💾 Space: O(1) 原地归一化
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def group_norm(
x: np.ndarray,
gamma: np.ndarray,
beta: np.ndarray,
num_groups: int = 4,
eps: float = 1e-5
) -> np.ndarray:
"""
GroupNorm 工业纯手写实现。
参数:
x: (B, C, H, W)
gamma, beta: (1, C, 1, 1) 可学习参数
num_groups: 分组数 G,要求 C % G == 0
"""
B, C, H, W = x.shape
assert C % num_groups == 0, f"通道数 {C} 必须能被组数 {num_groups} 整除"
channels_per_group =
# 核心步骤:重塑为 (B, G, C//G, H, W)
x_reshaped = x.reshape(B, num_groups, channels_per_group, H, W)
# 沿组内轴 (2, 3, 4) 求均值与方差
mean = np.mean(x_reshaped, axis=(2, 3, 4), keepdims=True)
var = np.var(x_reshaped, axis=(2, 3, 4), keepdims=True)
# 组内归一化
x_norm = (x_reshaped - mean) / np.sqrt(var + eps)
# 还原形状回 (B, C, H, W)
x_norm = x_norm.reshape(B, C, H, W)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why do diffusion U-Nets commonly prefer GroupNorm over LayerNorm?
Next: Inverted Dropout with Train/Eval Scaling →
💡

Core Mental Anchor / Mnemonic

Reshape channels into G groups, normalize across (C/G, H, W) per sample regardless of batch size.

📐 Mathematical Derivation & Core Formula

x∈RB×C×H×W→Reshape(B,G,C/G,H,W)  ⟹  μ,σ over (C/G,H,W)x \in \mathbb{R}^{B \times C \times H \times W} \xrightarrow{\text{Reshape}} (B, G, C/G, H, W) \implies \mu, \sigma \text{ over } (C/G, H, W)
### Channel Grouping & Special Cases
When batch sizes are small (e.g., 1 or 2 in object detection, segmentation, or high-resolution image synthesis), BatchNorm batch statistics become noisy and destabilize training.
GroupNorm (Wu & He, 2018):
- Divides CC channels into GG groups of C/GC/G channels each;
- Computes mean and variance over (C/G,H,W)(C/G, H, W) within each sample;
- Special case $G = 1$: Reduces to LayerNorm over all channels;
- Special case $G = C$: Reduces to InstanceNorm per channel (common in style transfer).

🔄 Tensor Dimensions & Shape Flow

(B, C, H, W) -> reshape (B, G, C//G, H, W) -> mean/var on (2,3,4) -> normalize -> reshape (B, C, H, W) -> affine

🛡️ Numerical Stability & Pitfalls Checklist

  • Ensure channel count C is evenly divisible by num_groups G.
  • Diffusion U-Nets commonly replace BatchNorm with GroupNorm(32) for small-batch stability.
  • Behavior is identical in training and evaluation modes.

💻 Reference Code Implementation

import numpy as np

def group_norm(
    x: np.ndarray,
    gamma: np.ndarray,
    beta: np.ndarray,
    num_groups: int = 4,
    eps: float = 1e-5
) -> np.ndarray:
    """
    GroupNorm 工业纯手写实现。
    参数:
        x: (B, C, H, W)
        gamma, beta: (1, C, 1, 1) 可学习参数
        num_groups: 分组数 G,要求 C % G == 0
    """
    B, C, H, W = x.shape
    assert C % num_groups == 0, f"通道数 {C} 必须能被组数 {num_groups} 整除"
    channels_per_group = C // num_groups
    
    # 核心步骤:重塑为 (B, G, C//G, H, W)
    x_reshaped = x.reshape(B, num_groups, channels_per_group, H, W)
    
    # 沿组内轴 (2, 3, 4) 求均值与方差
    mean = np.mean(x_reshaped, axis=(2, 3, 4), keepdims=True)
    var = np.var(x_reshaped, axis=(2, 3, 4), keepdims=True)
    
    # 组内归一化
    x_norm = (x_reshaped - mean) / np.sqrt(var + eps)
    
    # 还原形状回 (B, C, H, W)
    x_norm = x_norm.reshape(B, C, H, W)
    
    return gamma * x_norm + beta

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.random.randn(2, 8, 4, 4)
gamma = np.ones((1, 8, 1, 1))
beta = np.zeros((1, 8, 1, 1))
out = group_norm(x, gamma, beta, num_groups=4)
assert out.shape == (2, 8, 4, 4)
# 验证每个样本内每个组的均值为 0
out_g = out.reshape(2, 4, 2, 4, 4)
assert np.allclose(out_g.mean(axis=(2, 3, 4)), 0.0, atol=1e-3)
print("✓ GroupNorm 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why do diffusion U-Nets commonly prefer GroupNorm over LayerNorm?
Image feature maps exhibit strong spatial locality and cross-channel correlation within semantic groups. GroupNorm lets channels within each group share normalization statistics while keeping channel groups independent, which empirically works well for spatial visual features.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: BatchNorm1d with Train vs Eval DistinctionBack to Part B & All 105Next: Inverted Dropout with Train/Eval Scaling →