M3-009M3: Deep Learning FoundationsWeight InitializationEasy
⚡Flashcard Drill➔Mastery:

Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-010 →
📐 Mathematical Definition
Xavier: Var=2fanin+fanout;Kaiming: Var=2fanin\text{Xavier}:\ \mathrm{Var}=\frac{2}{fan_{in}+fan_{out}};\qquad \text{Kaiming}:\ \mathrm{Var}=\frac{2}{fan_{in}}
⚡ One-Sentence Core Answer
Xavier sets Var(W)=2nin+nout\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}} for linear/Tanh activations; Kaiming sets Var(W)=2nin\text{Var}(W) = \frac{2}{n_{\text{in}}} to compensate for ReLU's half-space zeroing.

📌 Key Takeaways

  • •
    Xavier (Glorot): Var(W)=2nin+nout\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}} (or 1nin\frac{1}{n_{\text{in}}}), assumes zero-centered linear/tanh regime
  • •
    Kaiming (He): Var(W)=2nin\text{Var}(W) = \frac{2}{n_{\text{in}}}, doubles variance to counteract ReLU halving activation power
  • •
    LeakyReLU adjustment: Kaiming initialization scales variance by 21+a2\frac{2}{1 + a^2} for negative slope aa

📐 Mathematical Derivations

Mathematical Derivations:
① Xavier (Glorot & Bengio, 2010):
Let y=∑i=1ninwixiy = \sum_{i=1}^{n_{\text{in}}} w_i x_i. Assuming independent zero-mean weights and inputs: Var(y)=ninVar(W)Var(X)\text{Var}(y) = n_{\text{in}} \text{Var}(W) \text{Var}(X). To preserve forward activation variance Var(y)=Var(X)\text{Var}(y) = \text{Var}(X), we require Var(W)=1/nin\text{Var}(W) = 1/n_{\text{in}}.
To simultaneously preserve backward gradient variance: Var(W)=1/nout\text{Var}(W) = 1/n_{\text{out}}. Taking the harmonic mean yields: Var(W)=2nin+nout\text{Var}(W) = \frac{2}{n_{\text{in}} + n_{\text{out}}}, with uniform bounds W∼U(−6nin+nout,6nin+nout)W \sim U\left(-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}, \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right).
② Kaiming / He (He et al., 2015):
For ReLU y=max⁡(0,z)y = \max(0, z), assuming symmetric distribution around 0, exactly half the inputs are set to zero: E[y2]=12E[z2]=12ninVar(W)E[x2]\mathbb{E}[y^2] = \frac{1}{2} \mathbb{E}[z^2] = \frac{1}{2} n_{\text{in}} \text{Var}(W) \mathbb{E}[x^2].
To maintain E[y2]=E[x2]\mathbb{E}[y^2] = \mathbb{E}[x^2], we must set: Var(W)=2nin\text{Var}(W) = \frac{2}{n_{\text{in}}}. Normal distribution: W∼N(0,2nin)W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{\text{in}}}}\right).

🏭 Production Trade-offs

Selection matrix: Use Xavier for symmetric activations with unit gradient at origin (Tanh, Sigmoid, Softsign, linear projections). Use Kaiming for rectified non-linearities (ReLU, LeakyReLU, PReLU, ELU).
⚠️ Common Interview Pitfalls
  • ✕
    Using Xavier initialization for deep ReLU networks, which leads to exponential activation decay and severe underflow in deep layers
  • ✕
    Using default standard normal initialization N(0,1)\mathcal{N}(0, 1), causing explosive gradients within the first 3 layers
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Xavier initialization fail in networks with 30+ layers when paired with ReLU activations?
  • ?
    How is ninn_{\text{in}} (fan_in) defined for 2D convolutional weight tensors of shape (Cout,Cin,Kh,Kw)(C_{\text{out}}, C_{\text{in}}, K_h, K_w)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-008: Why All-Zero Weight Initialization Fails in Neural Networks📋 Back to BankNext →M3-010: The 'Variance Preservation' Principle and Its Relation to Gradient Stability