M3-010M3: Deep Learning FoundationsWeight InitializationMedium
⚡Flashcard Drill➔Mastery:

The 'Variance Preservation' Principle and Its Relation to Gradient Stability

📐 Mathematical Definition
Var(h(l))≈Var(h(l−1))\mathrm{Var}(h^{(l)})\approx\mathrm{Var}(h^{(l-1)})
⚡ One-Sentence Core Answer
Maintaining unit activation and gradient variance across successive layers prevents exponential signal explosion or vanishing in deep networks.

📌 Key Takeaways

  • •
    Variance dynamics: Var(h(l))=[ninVar(W)]lVar(h(0))\text{Var}(h^{(l)}) = \left[ n_{\text{in}} \text{Var}(W) \right]^l \text{Var}(h^{(0)}); requires contraction factor ≈1\approx 1
  • •
    Exponential decay/explosion: factor <1< 1 leads to signal vanishing; factor >1> 1 leads to saturation and explosive gradients
  • •
    Modern relief: residual connections (y=x+F(x)y = x + F(x)) and normalization layers (LN/RMSNorm) guarantee variance stability

📐 Mathematical Derivations

Mathematical Dynamics: Consider a deep linear network without bias: h(l)=W(l)h(l−1)h^{(l)} = W^{(l)} h^{(l-1)}.
Under independence assumptions: Var(hj(l))=ninVar(W)Var(h(l−1))=[ninVar(W)]lVar(h(0))\text{Var}(h_j^{(l)}) = n_{\text{in}} \text{Var}(W) \text{Var}(h^{(l-1)}) = \left[ n_{\text{in}} \text{Var}(W) \right]^l \text{Var}(h^{(0)}).
- If ninVar(W)=1.1n_{\text{in}} \text{Var}(W) = 1.1, after 50 layers variance is multiplied by 1.150≈117.41.1^{50} \approx 117.4. In networks with bounded activations (Tanh), this saturates neurons into vanishing-gradient regimes.
- If ninVar(W)=0.9n_{\text{in}} \text{Var}(W) = 0.9, after 50 layers variance is 0.950≈0.005150.9^{50} \approx 0.00515. Signals evaporate to zero.
During backpropagation, gradients propagate via transposed weight matrices: δ(l−1)=(W(l))Tδ(l)\delta^{(l-1)} = (W^{(l)})^T \delta^{(l)}, multiplying gradient variance by [noutVar(W)]\left[ n_{\text{out}} \text{Var}(W) \right] at each step. Maintaining ninVar(W)≈1n_{\text{in}} \text{Var}(W) \approx 1 and noutVar(W)≈1n_{\text{out}} \text{Var}(W) \approx 1 guarantees both signal and gradient stability.

🏭 Production Trade-offs

Architectural evolution: In plain feedforward networks, precise initialization was mandatory to enable depth >10>10. Modern architectures relax this requirement: residual shortcuts ensure Var(x+F(x))≥Var(x)\text{Var}(x + F(x)) \ge \text{Var}(x), preventing vanishing, while LayerNorm actively normalizes variance to 1.0 at every block.
⚠️ Common Interview Pitfalls
  • ✕
    Ignoring activation function impact on variance (e.g., forgetting that ReLU zeros out half of the signal energy)
  • ✕
    Failing to inspect per-layer activation statistics when training deep custom architectures without normalization layers
🎯 Interviewer Follow-up Inquiries
  • ?
    How does a residual skip connection modify the variance accumulation formula across LL stacked layers?
  • ?
    Why does Pre-LayerNorm naturally enforce variance preservation compared to Post-LayerNorm?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

Explore the comprehensive technical article, exam cards, and global architecture tree.

← PreviousM3-009: Xavier and Kaiming Initialization: Mathematical Variances and Suitable Activations📋Back to BankNext →M3-011: Common Initialization Techniques in Transformers
中文速览与面试要点

解释“方差保持”原则,以及它与梯度稳定的关系

若每层输出方差被放大或缩小,深层会指数爆炸/衰减;保持方差≈1 使梯度尺度稳定。

核心要点

  • 为什么方差会逐层变化:设第 l 层的输出 h⁽ˡ⁾=σ(W⁽ˡ⁾h⁽ˡ⁻¹⁾),若忽略激活的非线性,Var(h⁽ˡ⁾)=fan_in·Var(w)·Var(h⁽ˡ⁻¹⁾)。若 fan_in·Var(w)>1,方差逐层 指数放大 (信号爆炸、饱和、梯度消失);若 指数衰减 (信号消失、深层学不到东西)。 方差保持原则 要求 fan_in·Var(w)≈1(考虑激活后相应调整),使信号与梯度在层间保持稳定尺度。 与梯度的关系 :反向传播的梯度也按同样的因子连乘,故方差保持同时保证了 梯度的尺度稳定 ——这是深层网络可训练的前提。 残差连接如何放松该要求 :y=x+F(x) 的方差为 Var(x)+Var(F)(若独立),即使 F 的方差很小,恒等路径也保证信号不衰减——这使残差网络的初始化要求大幅降低(可用更小的初始化,如 1/√(2L) 缩放)。 归一化层 则直接强制每层输入的均值方差(BN/LN),使方差保持成为'自动满足'。
  • 实践要点:① 诊断方法 ——初始化后跑一次前向,逐层打印激活的均值与方差(或用工具如 torchinfo );若某层方差偏离 1 超过一个数量级,说明初始化或架构有问题。② 与残差的叠加 ——残差网络中,残差分支的方差应 小于 恒等路径(否则累积爆炸);GPT-2 的 1/√(2L) 缩放正是使每层残差贡献的方差为 1/(2L),L 层累积后总方差≈1。③ 与深度的关系 ——网络越深,方差偏离的累积效应越严重(指数级);故深层网络对初始化更敏感,这也是 ResNet(残差)+ BN 组合能训练上千层的原因。④ 理论延伸 —— 动力等距性(dynamical isometry) 理论要求每层的雅可比奇异值都接近 1,此时梯度可无损传播(不爆炸不消失);正交初始化(RNN)与精心设计的缩放(μP)都是为此。⑤ μP(最大更新参数化) ——进一步要求'每层的更新幅度与宽度无关',使超参(学习率)可跨宽度迁移;这比单纯的方差保持更强。⑥ 实践优先级 ——(a) 有归一化层时,初始化的重要性降低(但仍有影响);(b) 无归一化时,初始化是训练成功的关键;(c) Transformer 因残差累积,必须用缩放初始化。
  • 这是 Xavier/Kaiming 的统一动机
  • 残差与归一化进一步放松该要求

常见陷阱

  • 忽略激活函数对方差的影响(ReLU 需加倍)
  • 深层网络不检查逐层激活方差