M3-061M3: Deep Learning FoundationsGradient Vanishing & ExplosionHard
⚡Flashcard Drill➔Mastery:
Gradient Scale Mismatch in Embedding and Output Layers: Logit Scaling and Tying
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Output logits grow with hidden dimension , saturating softmax gradients and creating extreme gradient scale discrepancies between embeddings and deep layers; resolve via logit scaling () or soft-capping.
📌 Key Takeaways
- •Variance growth: output logit has variance , growing with hidden dimension
- •Softmax gradient saturation: large logits push softmax probabilities to 0 and 1, causing gradients
- •Mitigations: scale logits by (Vaswani), apply logit soft-capping (Gemma), or adjust embedding learning rates under
📐 Mathematical Derivations
Mathematical Mechanics:
Let hidden state have zero mean and unit variance per feature: . Let output projection matrix be with weight variance .
The logit for vocabulary token is: .
The variance of the logit is: .
- If is initialized with standard variance , .
- However, in weight-tied models (), token embeddings require large norm to preserve semantic distances, causing to scale as . When , unconstrained logits reach magnitudes , completely saturating the softmax function.
Logit Soft-Capping (Gemma, Team et al., 2024):
Instead of raw logits, pass logits through a bounded hyperbolic tangent:
, where or .
This strictly bounds maximum logit magnitude to , preventing numerical overflow and softmax saturation without altering relative ranking.
Let hidden state have zero mean and unit variance per feature: . Let output projection matrix be with weight variance .
The logit for vocabulary token is: .
The variance of the logit is: .
- If is initialized with standard variance , .
- However, in weight-tied models (), token embeddings require large norm to preserve semantic distances, causing to scale as . When , unconstrained logits reach magnitudes , completely saturating the softmax function.
Logit Soft-Capping (Gemma, Team et al., 2024):
Instead of raw logits, pass logits through a bounded hyperbolic tangent:
, where or .
This strictly bounds maximum logit magnitude to , preventing numerical overflow and softmax saturation without altering relative ranking.
🏭 Production Trade-offs
Weight tying trade-offs: Tying embeddings saves substantial parameter memory ( parameters), but couples input representation learning with output probability separation. Untying weights allows independent scaling and optimizer learning rates.
⚠️ Common Interview Pitfalls
- ✕Training models with tied embeddings without scaling input tokens by or normalizing output logits, causing gradient underflow
- ✕Setting logit soft-capping threshold too low (), which prevents the model from generating confident predictions
🎯 Interviewer Follow-up Inquiries
- ?How does Gemma's logit soft-capping mathematically eliminate attention logit overflow?
- ?Why did modern multi-billion parameter LLMs shift away from weight tying toward untied embedding matrices?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚