Weight Initialization

Weight initialization chooses a distribution for the weights before training, aiming to keep the variance of forward activations and backward gradients preserved across layers (neither shrinking nor exploding), avoiding vanishing/exploding gradients. Zero or identical initialization is the anti-pattern: all neurons in a layer share the same weights → identical outputs and gradients → symmetry never breaks and every neuron learns the same function (equivalent to one neuron). Xavier/Glorot (2010, for the near-linear regime of tanh/sigmoid): forward variance preservation requires $\text{Var}(w) = 1/d_{\text{in}}$, backward $\text{Var}(w) = 1/d_{\text{out}}$; the compromise is $\text{Var}(w) = \frac{2}{d_{\text{in}} + d_{\text{out}}}$; uniform form $w \sim U\left[-\sqrt{\frac{6}{d_{\text{in}}+d_{\text{out}}}}, \sqrt{\frac{6}{d_{\text{in}}+d_{\text{out}}}}\right]$ (since $U[-a,a]$ has variance $a^2/3$, setting $a^2/3 = 2/(d_{\text{in}}+d_{\text{out}})$ gives $a = \sqrt{6/(d_{\text{in}}+d_{\text{out}})}$), or normal $N(0, 2/(d_{\text{in}}+d_{\text{out}}))$. Kaiming/He (2015, for ReLU): $\text{Var}(w) = \frac{2}{d_{\text{in}}}$, uniform $U\left[-\sqrt{\frac{6}{d_{\text{in}}}}, \sqrt{\frac{6}{d_{\text{in}}}}\right]$ or normal $N(0, 2/d_{\text{in}})$; PReLU generalizes to $\text{Var}(w) = \frac{2}{(1+a^2)d_{\text{in}}}$ ($a$ = negative-slope coefficient). Core derivation (assuming $E[w]=0$ and $x$ independent of $w$): for one output $y = \sum_i w_i x_i$, $\text{Var}(y) = d_{\text{in}}\,\text{Var}(w)\,\text{Var}(x)$; setting output variance equal to input variance gives $\text{Var}(w) = 1/d_{\text{in}}$, and ReLU halves the activation energy (zeroing the negative half), hence the factor 2.

the first line of defense in training any deep network (CNN/Transformer), complementary to BatchNorm (BN mitigates but does not fully fix a bad initialization); interview favorites: why zero-init fails, Xavier vs Kaiming selection, and how initialization relates to vanishing/exploding gradients.

weights too large → activation saturation (tanh derivative → 0 at the tails) or forward explosion; weights too small → exponential decay of signals (on the order of $0.9^{50} \approx 0.005$ after 50 layers). Variance-preserving initialization keeps signal magnitudes at $O(1)$ throughout, making dozens-to-hundreds-of-layer networks trainable. Note initialization only sets the starting point: long-term control of vanishing/exploding gradients also needs skip connections, BN/LN and gating — initialization complements, not replaces, them.

Why can't all weights be initialized to 0 (or the same value)? Explain the symmetry problem: identical initialization → identical gradients → symmetry never breaks → equivalent to a single neuron.

Derive Xavier's forward condition: from $\text{Var}(y) = d\,\text{Var}(w)\,\text{Var}(x)$ and variance preservation get $\text{Var}(w) = 1/d_{\text{in}}$; why does the final formula average as $2/(d_{\text{in}}+d_{\text{out}})$ (compromise between $1/d_{\text{in}}$ forward and $1/d_{\text{out}}$ backward)?

Write Xavier and Kaiming formulas and their activation targets: tanh/sigmoid → Xavier ($2/(d_{\text{in}}+d_{\text{out}})$), ReLU → Kaiming ($2/d_{\text{in}}$); why (ReLU zeroes half → factor 2)?

Where does the uniform bound $\sqrt{6/(d_{\text{in}}+d_{\text{out}})}$ come from? Solve $a^2/3 = 2/(d_{\text{in}}+d_{\text{out}})$ for $U[-a,a]$.

Initialization vs vanishing/exploding gradients: how does the initial variance determine signal propagation (magnitudes of forward activations and backward gradients); why are initialization and BN complementary?