M3-010M3: Deep Learning FoundationsWeight InitializationMedium
⚡Flashcard Drill➔Mastery:
The 'Variance Preservation' Principle and Its Relation to Gradient Stability
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Maintaining unit activation and gradient variance across successive layers prevents exponential signal explosion or vanishing in deep networks.
📌 Key Takeaways
- •Variance dynamics: ; requires contraction factor
- •Exponential decay/explosion: factor leads to signal vanishing; factor leads to saturation and explosive gradients
- •Modern relief: residual connections () and normalization layers (LN/RMSNorm) guarantee variance stability
📐 Mathematical Derivations
Mathematical Dynamics: Consider a deep linear network without bias: .
Under independence assumptions: .
- If , after 50 layers variance is multiplied by . In networks with bounded activations (Tanh), this saturates neurons into vanishing-gradient regimes.
- If , after 50 layers variance is . Signals evaporate to zero.
During backpropagation, gradients propagate via transposed weight matrices: , multiplying gradient variance by at each step. Maintaining and guarantees both signal and gradient stability.
Under independence assumptions: .
- If , after 50 layers variance is multiplied by . In networks with bounded activations (Tanh), this saturates neurons into vanishing-gradient regimes.
- If , after 50 layers variance is . Signals evaporate to zero.
During backpropagation, gradients propagate via transposed weight matrices: , multiplying gradient variance by at each step. Maintaining and guarantees both signal and gradient stability.
🏭 Production Trade-offs
Architectural evolution: In plain feedforward networks, precise initialization was mandatory to enable depth . Modern architectures relax this requirement: residual shortcuts ensure , preventing vanishing, while LayerNorm actively normalizes variance to 1.0 at every block.
⚠️ Common Interview Pitfalls
- ✕Ignoring activation function impact on variance (e.g., forgetting that ReLU zeros out half of the signal energy)
- ✕Failing to inspect per-layer activation statistics when training deep custom architectures without normalization layers
🎯 Interviewer Follow-up Inquiries
- ?How does a residual skip connection modify the variance accumulation formula across stacked layers?
- ?Why does Pre-LayerNorm naturally enforce variance preservation compared to Post-LayerNorm?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚