M3-025M3: Deep Learning FoundationsNormalization TechniquesMedium
⚡Flashcard Drill➔Mastery:

Group Normalization (GN): Mechanism and Applicable Scenarios

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-026 →
📐 Mathematical Definition
GN: μ,σ computed over (C/G, H, W) per sample\text{GN}:\ \mu,\sigma\ \text{computed over (C/G, H, W) per sample}
⚡ One-Sentence Core Answer
Group Normalization divides channels into groups and normalizes within each group per sample; being batch-independent, it is ideal for memory-heavy vision tasks like object detection.

📌 Key Takeaways

  • •
    Mechanism: divides CC channels into GG groups (each with C/GC/G channels) and computes μ,σ\mu, \sigma across (group channels ×\times height ×\times width)
  • •
    Batch independence: normalizes each image independently, eliminating small-batch degradation
  • •
    Special cases: G=1G=1 reduces to LayerNorm; G=CG=C reduces to InstanceNorm

📐 Mathematical Derivations

Mathematical Formulation (Wu & He, ECCV 2018):
Let input feature map be x∈RN×C×H×Wx \in \mathbb{R}^{N \times C \times H \times W}. Divide channels into GG groups (default G=32G=32). For each group g∈[1,G]g \in [1, G], the channel index set is Sg={c∣⌊c⋅GC⌋=g}\mathcal{S}_g = \left\{ c \mid \left\lfloor \frac{c \cdot G}{C} \right\rfloor = g \right\}.
For each instance ii and group gg, compute mean and variance across Sg×H×W\mathcal{S}_g \times H \times W:
μi,g=1(C/G)HW∑c∈Sg∑h=1H∑w=1Wxi,c,h,w\mu_{i, g} = \frac{1}{(C/G) H W} \sum_{c \in \mathcal{S}_g} \sum_{h=1}^H \sum_{w=1}^W x_{i, c, h, w},
σi,g2=1(C/G)HW∑c∈Sg∑h=1H∑w=1W(xi,c,h,w−μi,g)2\sigma_{i, g}^2 = \frac{1}{(C/G) H W} \sum_{c \in \mathcal{S}_g} \sum_{h=1}^H \sum_{w=1}^W (x_{i, c, h, w} - \mu_{i, g})^2.
Normalize: x^i,c,h,w=xi,c,h,w−μi,gσi,g2+ϵγc+βc\hat{x}_{i, c, h, w} = \frac{x_{i, c, h, w} - \mu_{i, g}}{\sqrt{\sigma_{i, g}^2 + \epsilon}} \gamma_c + \beta_c.
Relationship with other norms:
- When G=1G=1, all channels form 1 group: GN ≡\equiv LayerNorm (LN).
- When G=CG=C, each group has 1 channel: GN ≡\equiv InstanceNorm (IN).

🏭 Production Trade-offs

Target application scenarios: Object detection (Mask R-CNN, Faster R-CNN), semantic segmentation, and video recognition where high input resolutions (1024×10241024 \times 1024) restrict batch size per GPU to 1 or 2 images. While BatchNorm breaks down at N≤2N \le 2, Group Normalization maintains consistent accuracy invariant to batch size.
⚠️ Common Interview Pitfalls
  • ✕
    Setting group count G=1G=1, which collapses GN into LayerNorm and erases channel-specific spatial representations in vision backbones
  • ✕
    Using GroupNorm in standard ImageNet classification with large batches (B=256B=256), where BatchNorm is slightly faster due to optimized cuDNN kernels
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Group Normalization perform significantly better on vision tasks than LayerNorm (G=1G=1)?
  • ?
    How does channel grouping in GN relate to classical computer vision representation concepts like SIFT and HOG?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-024: Side Effects of Normalization Layers and Alternative Architectures📋 Back to BankNext →M3-026: Weight Standardization and Its Role in Normalizer-Free Networks