M3-061M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:

Gradient Scale Mismatch in Embedding and Output Layers: Logit Scaling and Tying

📐 Mathematical Definition
logits=hW⊤;Var(logit)∝d ⇒ requires W∼N(0,1/d) or logit scaling\text{logits}=hW^{\top};\quad \mathrm{Var}(\text{logit})\propto d\ \Rightarrow\ \text{requires}\ W\sim\mathcal{N}(0,1/d)\ \text{or logit scaling}
⚡ One-Sentence Core Answer
Output logits grow with hidden dimension dd, saturating softmax gradients and creating extreme gradient scale discrepancies between embeddings and deep layers; resolve via logit scaling (1/d1/\sqrt{d}) or soft-capping.

📌 Key Takeaways

  • •
    Variance growth: output logit z=hWoutTz = h W_{\text{out}}^T has variance ∝d⋅Var(W)\propto d \cdot \text{Var}(W), growing with hidden dimension
  • •
    Softmax gradient saturation: large logits push softmax probabilities to 0 and 1, causing gradients pi−yi→0p_i - y_i \to 0
  • •
    Mitigations: scale logits by 1/d1/\sqrt{d} (Vaswani), apply logit soft-capping (Gemma), or adjust embedding learning rates under μP\mu P

📐 Mathematical Derivations

Mathematical Mechanics:
Let hidden state h∈Rdh \in \mathbb{R}^d have zero mean and unit variance per feature: E[hi]=0,Var(hi)=1\mathbb{E}[h_i] = 0, \text{Var}(h_i) = 1. Let output projection matrix be Wout∈RV×dW_{\text{out}} \in \mathbb{R}^{V \times d} with weight variance σw2\sigma_w^2.
The logit for vocabulary token kk is: zk=∑i=1dWk,ihiz_k = \sum_{i=1}^d W_{k, i} h_i.
The variance of the logit is: Var(zk)=d⋅σw2\text{Var}(z_k) = d \cdot \sigma_w^2.
- If WoutW_{\text{out}} is initialized with standard variance σw2∼1/d\sigma_w^2 \sim 1/d, Var(zk)≈1\text{Var}(z_k) \approx 1.
- However, in weight-tied models (Wout=WembedW_{\text{out}} = W_{\text{embed}}), token embeddings require large norm to preserve semantic distances, causing Var(zk)\text{Var}(z_k) to scale as O(d)O(d). When d=8192d = 8192, unconstrained logits reach magnitudes >100> 100, completely saturating the softmax function.
Logit Soft-Capping (Gemma, Team et al., 2024):
Instead of raw logits, pass logits through a bounded hyperbolic tangent:
zcapped=C⋅tanh⁡(zC)z_{\text{capped}} = C \cdot \tanh\left( \frac{z}{C} \right), where C=30.0C = 30.0 or 50.050.0.
This strictly bounds maximum logit magnitude to [−C,C][-C, C], preventing numerical overflow and softmax saturation without altering relative ranking.

🏭 Production Trade-offs

Weight tying trade-offs: Tying embeddings saves substantial parameter memory (V×dV \times d parameters), but couples input representation learning with output probability separation. Untying weights allows independent scaling and optimizer learning rates.
⚠️ Common Interview Pitfalls
  • ✕
    Training models with tied embeddings without scaling input tokens by d\sqrt{d} or normalizing output logits, causing gradient underflow
  • ✕
    Setting logit soft-capping threshold CC too low (<10< 10), which prevents the model from generating confident predictions
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Gemma's logit soft-capping mathematically eliminate attention logit overflow?
  • ?
    Why did modern multi-billion parameter LLMs shift away from weight tying toward untied embedding matrices?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

Explore the comprehensive technical article, exam cards, and global architecture tree.

← PreviousM3-060: The Loss Spike Phenomenon: Gradient Norm Correlations, Root Causes, and Mitigations📋Back to BankNext →M3-062: Per-Layer and Adaptive Gradient Clipping (AGC) Explained
中文速览与面试要点

解释 embedding 与输出层的梯度尺度问题(logit 缩放)

输出层 logit 随隐藏维度增大而增大,导致 softmax 梯度饱和、embedding 梯度失衡;用 logit 缩放/μP 修正。

核心要点

  • 数学机理:设隐藏向量 h∈ℝ^d(各分量方差约 1)、输出权重 W∈ℝ^V×d,则 logit z=hW^T 的每个分量是 d 个独立项之和,方差 ∝ d·Var(W)。若 W 用标准初始化(Var(W)=1/d),则 Var(z)≈1(合理);但若用 Var(W)=1/√d 或 1(不随 d 缩放),则 Var(z)∝√d 或 d—— logit 幅度随隐藏维度增长 ,使 softmax 进入饱和区(最大 logit 主导、其余概率趋 0),进而 (a) softmax 的梯度趋 0(饱和)、(b) 输出层与 embedding 层收到异常尺度的梯度。此外, 输出层梯度 ∂L/∂W_out=(p−y)h^T 的尺度取决于 (p−y)(softmax 误差); embedding 层梯度 只在出现的 token 上非零(稀疏),其有效范数天然小于稠密层——若二者权重共享(tied embedding),则两组梯度叠加,尺度更复杂。 μP 的修正 :在最大更新参数化下,输出层权重应初始化为 1/d(而非 1/√d),且 logits 需乘一个 1/d 或 1/√d 的缩放,使 logit 方差与宽度无关。 logit soft-capping (Gemma-2)则直接把 logit 限制在 [−c, c],防止极端 logit 主导。
  • 深度剖析与工程权衡:① 为什么不缩放会出问题 ——logit 幅度大 → softmax 接近 one-hot → 梯度饱和、训练不稳;同时 attention logits 过大会使注意力过度集中('attention entropy 崩塌'),损害长上下文能力。② 注意力的对应问题 ——注意力分数 q·k/√d_k 中的 1/√d_k 缩放正是同一问题的修正:若不除 √d_k,logit 方差 ∝d_k、softmax 饱和;故 QK-Norm 或 1/√d_k 是必需项。③ μP 的完整规则 ——隐藏层用 1/√fan_in(标准)、输出层用 1/fan_in、输入层(embedding)用 1(不缩放)、lr 随宽度反向缩放;这套规则使超参可跨宽度迁移,是'大模型超参可预测'的理论基础。④ tied embedding 的权衡 ——共享输入 embedding 与输出层可省参数量(V×d)并提升小模型效果,但使两组梯度耦合;大模型常不共享(因 V 大、共享会限制表达)。⑤ 实践诊断 ——记录 logits 的绝对最大值与 softmax 熵:若熵持续下降趋于 0,说明 logit 失控。⑥ 面试要点 ——被问'为什么 attention 要除以 √d',答'控制 logit 方差与宽度无关、避免 softmax 饱和';若进一步追问'输出层呢',答 μP 的 1/d 缩放,展示深度理解。
  • 初始化不当会使 logit 幅度 ∝√d,softmax 饱和
  • 输出层与 embedding 层梯度尺度常与中间层差 1~2 个量级
  • μP 给出输出层应 1/d(而非 1/√d)的缩放

常见陷阱

  • 认为 1/√d_k 只是经验技巧(实为方差控制)
  • 忽略 tied embedding 对梯度尺度的影响