M3-061M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:

Gradient Scale Mismatch in Embedding and Output Layers: Logit Scaling and Tying

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-062 →
📐 Mathematical Definition
logits=hW⊤;Var(logit)∝d ⇒ requires W∼N(0,1/d) or logit scaling\text{logits}=hW^{\top};\quad \mathrm{Var}(\text{logit})\propto d\ \Rightarrow\ \text{requires}\ W\sim\mathcal{N}(0,1/d)\ \text{or logit scaling}
⚡ One-Sentence Core Answer
Output logits grow with hidden dimension dd, saturating softmax gradients and creating extreme gradient scale discrepancies between embeddings and deep layers; resolve via logit scaling (1/d1/\sqrt{d}) or soft-capping.

📌 Key Takeaways

  • •
    Variance growth: output logit z=hWoutTz = h W_{\text{out}}^T has variance ∝d⋅Var(W)\propto d \cdot \text{Var}(W), growing with hidden dimension
  • •
    Softmax gradient saturation: large logits push softmax probabilities to 0 and 1, causing gradients pi−yi→0p_i - y_i \to 0
  • •
    Mitigations: scale logits by 1/d1/\sqrt{d} (Vaswani), apply logit soft-capping (Gemma), or adjust embedding learning rates under μP\mu P

📐 Mathematical Derivations

Mathematical Mechanics:
Let hidden state h∈Rdh \in \mathbb{R}^d have zero mean and unit variance per feature: E[hi]=0,Var(hi)=1\mathbb{E}[h_i] = 0, \text{Var}(h_i) = 1. Let output projection matrix be Wout∈RV×dW_{\text{out}} \in \mathbb{R}^{V \times d} with weight variance σw2\sigma_w^2.
The logit for vocabulary token kk is: zk=∑i=1dWk,ihiz_k = \sum_{i=1}^d W_{k, i} h_i.
The variance of the logit is: Var(zk)=d⋅σw2\text{Var}(z_k) = d \cdot \sigma_w^2.
- If WoutW_{\text{out}} is initialized with standard variance σw2∼1/d\sigma_w^2 \sim 1/d, Var(zk)≈1\text{Var}(z_k) \approx 1.
- However, in weight-tied models (Wout=WembedW_{\text{out}} = W_{\text{embed}}), token embeddings require large norm to preserve semantic distances, causing Var(zk)\text{Var}(z_k) to scale as O(d)O(d). When d=8192d = 8192, unconstrained logits reach magnitudes >100> 100, completely saturating the softmax function.
Logit Soft-Capping (Gemma, Team et al., 2024):
Instead of raw logits, pass logits through a bounded hyperbolic tangent:
zcapped=C⋅tanh⁡(zC)z_{\text{capped}} = C \cdot \tanh\left( \frac{z}{C} \right), where C=30.0C = 30.0 or 50.050.0.
This strictly bounds maximum logit magnitude to [−C,C][-C, C], preventing numerical overflow and softmax saturation without altering relative ranking.

🏭 Production Trade-offs

Weight tying trade-offs: Tying embeddings saves substantial parameter memory (V×dV \times d parameters), but couples input representation learning with output probability separation. Untying weights allows independent scaling and optimizer learning rates.
⚠️ Common Interview Pitfalls
  • ✕
    Training models with tied embeddings without scaling input tokens by d\sqrt{d} or normalizing output logits, causing gradient underflow
  • ✕
    Setting logit soft-capping threshold CC too low (<10< 10), which prevents the model from generating confident predictions
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Gemma's logit soft-capping mathematically eliminate attention logit overflow?
  • ?
    Why did modern multi-billion parameter LLMs shift away from weight tying toward untied embedding matrices?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-060: The Loss Spike Phenomenon: Gradient Norm Correlations, Root Causes, and Mitigations📋 Back to BankNext →M3-062: Per-Layer and Adaptive Gradient Clipping (AGC) Explained