M3-009M3: Deep Learning FoundationsWeight InitializationEasy
⚡Flashcard Drill➔Mastery:
Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Xavier sets for linear/Tanh activations; Kaiming sets to compensate for ReLU's half-space zeroing.
📌 Key Takeaways
- •Xavier (Glorot): (or ), assumes zero-centered linear/tanh regime
- •Kaiming (He): , doubles variance to counteract ReLU halving activation power
- •LeakyReLU adjustment: Kaiming initialization scales variance by for negative slope
📐 Mathematical Derivations
Mathematical Derivations:
① Xavier (Glorot & Bengio, 2010):
Let . Assuming independent zero-mean weights and inputs: . To preserve forward activation variance , we require .
To simultaneously preserve backward gradient variance: . Taking the harmonic mean yields: , with uniform bounds .
② Kaiming / He (He et al., 2015):
For ReLU , assuming symmetric distribution around 0, exactly half the inputs are set to zero: .
To maintain , we must set: . Normal distribution: .
① Xavier (Glorot & Bengio, 2010):
Let . Assuming independent zero-mean weights and inputs: . To preserve forward activation variance , we require .
To simultaneously preserve backward gradient variance: . Taking the harmonic mean yields: , with uniform bounds .
② Kaiming / He (He et al., 2015):
For ReLU , assuming symmetric distribution around 0, exactly half the inputs are set to zero: .
To maintain , we must set: . Normal distribution: .
🏭 Production Trade-offs
Selection matrix: Use Xavier for symmetric activations with unit gradient at origin (Tanh, Sigmoid, Softsign, linear projections). Use Kaiming for rectified non-linearities (ReLU, LeakyReLU, PReLU, ELU).
⚠️ Common Interview Pitfalls
- ✕Using Xavier initialization for deep ReLU networks, which leads to exponential activation decay and severe underflow in deep layers
- ✕Using default standard normal initialization , causing explosive gradients within the first 3 layers
🎯 Interviewer Follow-up Inquiries
- ?Why does Xavier initialization fail in networks with 30+ layers when paired with ReLU activations?
- ?How is (fan_in) defined for 2D convolutional weight tensors of shape ?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚