M4-044M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:

Attention Entropy Collapse and How QK-Norm Resolves It

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-045 →
📐 Mathematical Definition
H(αi)=−∑jαijlog⁡αij→0;QK-Norm: Q,K←RMSNorm(⋅)H(\alpha_i)=-\sum_j\alpha_{ij}\log\alpha_{ij}\to0;\qquad \text{QK-Norm}:\ Q,K\leftarrow\mathrm{RMSNorm}(\cdot)
⚡ One-Sentence Core Answer
Entropy collapse occurs when unconstrained query-key magnitudes cause attention logits to explode, collapsing softmax into rigid one-hot distributions; QK-Norm bounds logits to preserve entropy.

📌 Key Takeaways

  • •
    Entropy collapse: Shannon entropy H(Ai)=−∑pjlog⁡pj→0H(A_i) = -\sum p_j \log p_j \to 0; attention focuses exclusively on 1 single token across all heads
  • •
    Gradient vanishing: one-hot softmax derivative p(1−p)→0p(1 - p) \to 0, freezing backpropagation through attention pathways
  • •
    QK-Norm remedy: applies RMSNorm to Queries and Keys, bounding logit magnitudes to ≤dk\le \sqrt{d_k} and sustaining healthy entropy

📐 Mathematical Derivations

Mathematical Mechanics (Dehghani et al., ViT-22B 2023; Henry et al., 2020):
In deep models, weight norms ∥WQ∥,∥WK∥\|W_Q\|, \|W_K\| grow monotonically during training. The attention logit is zij=qiTkjdkz_{ij} = \frac{q_i^T k_j}{\sqrt{d_k}}.
- If ∥qi∥2\|q_i\|_2 and ∥kj∥2\|k_j\|_2 grow from dk\sqrt{d_k} to 10dk10\sqrt{d_k}, logit magnitudes expand to ±100\pm 100.
- In softmax: pi=ezi∑ezjp_i = \frac{e^{z_i}}{\sum e^{z_j}}. When zmax⁡−zsecond>15z_{\max} - z_{\text{second}} > 15, pmax⁡≈1.0p_{\max} \approx 1.0 and all other pj≈0p_j \approx 0.
- Attention Entropy Collapse:
The attention entropy H(p)=−∑j=1Npjlog⁡pjH(p) = - \sum_{j=1}^N p_j \log p_j collapses toward 0.
- Gradient Vanishing:
The Jacobian of softmax is ∂pj∂zk=pj(δjk−pk)\frac{\partial p_j}{\partial z_k} = p_j (\delta_{jk} - p_k). When pp is a one-hot vector, the diagonal is 1(1−1)=01(1 - 1) = 0 and off-diagonals are 0(0−0)=00(0 - 0) = 0. Gradients cannot flow through the attention matrix, freezing model learning.
- The QK-Norm Fix:
Normalize: q~=RMSNorm(q),  k~=RMSNorm(k)\tilde{q} = \text{RMSNorm}(q), \; \tilde{k} = \text{RMSNorm}(k).
Then ∥q~i∥2=dk\|\tilde{q}_i\|_2 = \sqrt{d_k} and ∥k~j∥2=dk\|\tilde{k}_j\|_2 = \sqrt{d_k}.
The logit satisfies: ∣q~iTk~jdk∣=∣dkdkcos⁡(θ)dk∣=dk∣cos⁡(θ)∣≤dk\left| \frac{\tilde{q}_i^T \tilde{k}_j}{\sqrt{d_k}} \right| = \left| \frac{\sqrt{d_k} \sqrt{d_k} \cos(\theta)}{\sqrt{d_k}} \right| = \sqrt{d_k} |\cos(\theta)| \le \sqrt{d_k}.
When dk=64d_k = 64, logits are strictly bounded within [−8,8][-8, 8], guaranteeing healthy, non-vanishing attention entropy.

🏭 Production Trade-offs

Zero Overhead: QK-Norm adds simple element-wise root-mean-square normalization in registers, adding <0.5%<0.5\% compute overhead while eliminating one of the primary failure modes in scaling foundation models.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to fix attention entropy collapse by lowering learning rate, which slows training without solving unconstrained logit growth
  • ✕
    Applying QK-Norm without the 1/dk1/\sqrt{d_k} factor, causing under-confident attention distributions
🎯 Interviewer Follow-up Inquiries
  • ?
    How does tracking attention entropy (H(A)H(A)) during training serve as an early warning for loss spikes?
  • ?
    Why does the one-hot collapse of softmax attention destroy backpropagated gradients?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-043: Cross-Attention vs Self-Attention: Structural Differences and Multimodal Roles📋 Back to BankNext →M4-045: Alternatives to Softmax Attention: Sigmoid Attention and Normalization Variants