M3-023M3: Deep Learning FoundationsNormalization TechniquesMedium
⚡Flashcard Drill➔Mastery:
Why BatchNorm Uses Running Statistics During Inference
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Inference often runs on single samples (batch size = 1) where batch variance is zero, and predictions must be deterministic and independent of other co-batched inputs.
📌 Key Takeaways
- •Single sample availability: online inference frequently evaluates , where sample variance is undefined / zero
- •Determinism requirement: the output for sample must never depend on what other unrelated samples were batched with it
- •Running statistics: computes exponential moving averages of training batch means and variances:
📐 Mathematical Derivations
Mechanisms and Failure Modes:
- Training Mode: For mini-batch :
, .
Simultaneously updates running statistics via Exponential Moving Average (EMA) with momentum :
, .
- Inference Mode (`model.eval()`):
Freezes batch statistics and uses the global running estimates: .
- Why current batch statistics fail at inference: If an inference server receives , , causing division by zero. Furthermore, if a fraudulent transaction is batched with 3 normal transactions, its prediction would change depending on who it was batched with, violating system determinism and leaking batch information.
- Training Mode: For mini-batch :
, .
Simultaneously updates running statistics via Exponential Moving Average (EMA) with momentum :
, .
- Inference Mode (`model.eval()`):
Freezes batch statistics and uses the global running estimates: .
- Why current batch statistics fail at inference: If an inference server receives , , causing division by zero. Furthermore, if a fraudulent transaction is batched with 3 normal transactions, its prediction would change depending on who it was batched with, violating system determinism and leaking batch information.
🏭 Production Trade-offs
Weight folding optimization: During inference, because are constants, BatchNorm can be mathematically folded into the preceding Conv2D layer: , , reducing inference latency to zero additional FLOPs.
⚠️ Common Interview Pitfalls
- ✕Forgetting to call
model.eval(), which causes the model to compute running stats and normalize using test batch statistics - ✕Fine-tuning on small datasets without updating or freezing running statistics appropriately, causing distribution collapse
🎯 Interviewer Follow-up Inquiries
- ?How does folding BatchNorm into Conv2D weights eliminate normalization latency during production inference?
- ?What causes training and inference performance discrepancy when fine-tuning a pretrained model with frozen BatchNorm?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚