M3-010M3: Deep Learning FoundationsWeight InitializationMedium
⚡Flashcard Drill➔Mastery:

The 'Variance Preservation' Principle and Its Relation to Gradient Stability

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-011 →
📐 Mathematical Definition
Var(h(l))≈Var(h(l−1))\mathrm{Var}(h^{(l)})\approx\mathrm{Var}(h^{(l-1)})
⚡ One-Sentence Core Answer
Maintaining unit activation and gradient variance across successive layers prevents exponential signal explosion or vanishing in deep networks.

📌 Key Takeaways

  • •
    Variance dynamics: Var(h(l))=[ninVar(W)]lVar(h(0))\text{Var}(h^{(l)}) = \left[ n_{\text{in}} \text{Var}(W) \right]^l \text{Var}(h^{(0)}); requires contraction factor ≈1\approx 1
  • •
    Exponential decay/explosion: factor <1< 1 leads to signal vanishing; factor >1> 1 leads to saturation and explosive gradients
  • •
    Modern relief: residual connections (y=x+F(x)y = x + F(x)) and normalization layers (LN/RMSNorm) guarantee variance stability

📐 Mathematical Derivations

Mathematical Dynamics: Consider a deep linear network without bias: h(l)=W(l)h(l−1)h^{(l)} = W^{(l)} h^{(l-1)}.
Under independence assumptions: Var(hj(l))=ninVar(W)Var(h(l−1))=[ninVar(W)]lVar(h(0))\text{Var}(h_j^{(l)}) = n_{\text{in}} \text{Var}(W) \text{Var}(h^{(l-1)}) = \left[ n_{\text{in}} \text{Var}(W) \right]^l \text{Var}(h^{(0)}).
- If ninVar(W)=1.1n_{\text{in}} \text{Var}(W) = 1.1, after 50 layers variance is multiplied by 1.150≈117.41.1^{50} \approx 117.4. In networks with bounded activations (Tanh), this saturates neurons into vanishing-gradient regimes.
- If ninVar(W)=0.9n_{\text{in}} \text{Var}(W) = 0.9, after 50 layers variance is 0.950≈0.005150.9^{50} \approx 0.00515. Signals evaporate to zero.
During backpropagation, gradients propagate via transposed weight matrices: δ(l−1)=(W(l))Tδ(l)\delta^{(l-1)} = (W^{(l)})^T \delta^{(l)}, multiplying gradient variance by [noutVar(W)]\left[ n_{\text{out}} \text{Var}(W) \right] at each step. Maintaining ninVar(W)≈1n_{\text{in}} \text{Var}(W) \approx 1 and noutVar(W)≈1n_{\text{out}} \text{Var}(W) \approx 1 guarantees both signal and gradient stability.

🏭 Production Trade-offs

Architectural evolution: In plain feedforward networks, precise initialization was mandatory to enable depth >10>10. Modern architectures relax this requirement: residual shortcuts ensure Var(x+F(x))≥Var(x)\text{Var}(x + F(x)) \ge \text{Var}(x), preventing vanishing, while LayerNorm actively normalizes variance to 1.0 at every block.
⚠️ Common Interview Pitfalls
  • ✕
    Ignoring activation function impact on variance (e.g., forgetting that ReLU zeros out half of the signal energy)
  • ✕
    Failing to inspect per-layer activation statistics when training deep custom architectures without normalization layers
🎯 Interviewer Follow-up Inquiries
  • ?
    How does a residual skip connection modify the variance accumulation formula across LL stacked layers?
  • ?
    Why does Pre-LayerNorm naturally enforce variance preservation compared to Post-LayerNorm?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-009: Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations📋 Back to BankNext →M3-011: Common Initialization Techniques in Transformers