M4-044M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:
Attention Entropy Collapse and How QK-Norm Resolves It
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Entropy collapse occurs when unconstrained query-key magnitudes cause attention logits to explode, collapsing softmax into rigid one-hot distributions; QK-Norm bounds logits to preserve entropy.
📌 Key Takeaways
- •Entropy collapse: Shannon entropy ; attention focuses exclusively on 1 single token across all heads
- •Gradient vanishing: one-hot softmax derivative , freezing backpropagation through attention pathways
- •QK-Norm remedy: applies RMSNorm to Queries and Keys, bounding logit magnitudes to and sustaining healthy entropy
📐 Mathematical Derivations
Mathematical Mechanics (Dehghani et al., ViT-22B 2023; Henry et al., 2020):
In deep models, weight norms grow monotonically during training. The attention logit is .
- If and grow from to , logit magnitudes expand to .
- In softmax: . When , and all other .
- Attention Entropy Collapse:
The attention entropy collapses toward 0.
- Gradient Vanishing:
The Jacobian of softmax is . When is a one-hot vector, the diagonal is and off-diagonals are . Gradients cannot flow through the attention matrix, freezing model learning.
- The QK-Norm Fix:
Normalize: .
Then and .
The logit satisfies: .
When , logits are strictly bounded within , guaranteeing healthy, non-vanishing attention entropy.
In deep models, weight norms grow monotonically during training. The attention logit is .
- If and grow from to , logit magnitudes expand to .
- In softmax: . When , and all other .
- Attention Entropy Collapse:
The attention entropy collapses toward 0.
- Gradient Vanishing:
The Jacobian of softmax is . When is a one-hot vector, the diagonal is and off-diagonals are . Gradients cannot flow through the attention matrix, freezing model learning.
- The QK-Norm Fix:
Normalize: .
Then and .
The logit satisfies: .
When , logits are strictly bounded within , guaranteeing healthy, non-vanishing attention entropy.
🏭 Production Trade-offs
Zero Overhead: QK-Norm adds simple element-wise root-mean-square normalization in registers, adding compute overhead while eliminating one of the primary failure modes in scaling foundation models.
⚠️ Common Interview Pitfalls
- ✕Attempting to fix attention entropy collapse by lowering learning rate, which slows training without solving unconstrained logit growth
- ✕Applying QK-Norm without the factor, causing under-confident attention distributions
🎯 Interviewer Follow-up Inquiries
- ?How does tracking attention entropy () during training serve as an early warning for loss spikes?
- ?Why does the one-hot collapse of softmax attention destroy backpropagated gradients?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚