Weight Initialization

Weight initialization chooses a distribution for the weights before training, aiming to keep the variance of forward activations and backward gradients preserved across layers (neither shrinking nor exploding), avoiding vanishing/exploding gradients. Zero or identical initialization is the anti-pattern: all neurons in a layer share the same weights → identical outputs and gradients → symmetry never breaks and every neuron learns the same function (equivalent to one neuron). Xavier/Glorot (2010, for the near-linear regime of tanh/sigmoid): forward variance preservation requires $\text{Var}(w) = 1/d_{\text{in}}$, backward $\text{Var}(w) = 1/d_{\text{out}}$; the compromise is $\text{Var}(w) = \frac{2}{d_{\text{in}} + d_{\text{out}}}$; uniform form $w \sim U\left[-\sqrt{\frac{6}{d_{\text{in}}+d_{\text{out}}}}, \sqrt{\frac{6}{d_{\text{in}}+d_{\text{out}}}}\right]$ (since $U[-a,a]$ has variance $a^2/3$, setting $a^2/3 = 2/(d_{\text{in}}+d_{\text{out}})$ gives $a = \sqrt{6/(d_{\text{in}}+d_{\text{out}})}$), or normal $N(0, 2/(d_{\text{in}}+d_{\text{out}}))$. Kaiming/He (2015, for ReLU): $\text{Var}(w) = \frac{2}{d_{\text{in}}}$, uniform $U\left[-\sqrt{\frac{6}{d_{\text{in}}}}, \sqrt{\frac{6}{d_{\text{in}}}}\right]$ or normal $N(0, 2/d_{\text{in}})$; PReLU generalizes to $\text{Var}(w) = \frac{2}{(1+a^2)d_{\text{in}}}$ ($a$ = negative-slope coefficient). Core derivation (assuming $E[w]=0$ and $x$ independent of $w$): for one output $y = \sum_i w_i x_i$, $\text{Var}(y) = d_{\text{in}}\,\text{Var}(w)\,\text{Var}(x)$; setting output variance equal to input variance gives $\text{Var}(w) = 1/d_{\text{in}}$, and ReLU halves the activation energy (zeroing the negative half), hence the factor 2.

the first line of defense in training any deep network (CNN/Transformer), complementary to BatchNorm (BN mitigates but does not fully fix a bad initialization); interview favorites: why zero-init fails, Xavier vs Kaiming selection, and how initialization relates to vanishing/exploding gradients.

weights too large → activation saturation (tanh derivative → 0 at the tails) or forward explosion; weights too small → exponential decay of signals (on the order of $0.9^{50} \approx 0.005$ after 50 layers). Variance-preserving initialization keeps signal magnitudes at $O(1)$ throughout, making dozens-to-hundreds-of-layer networks trainable. Note initialization only sets the starting point: long-term control of vanishing/exploding gradients also needs skip connections, BN/LN and gating — initialization complements, not replaces, them.

Test Your Knowledge: Practice Questions for "Weight Initialization"

Single choice pitfall questions with instant feedback and mistake tracking.