M4-035M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Easy
⚡Flashcard Drill➔Mastery:
Multi-Query Attention (MQA) and Grouped-Query Attention (GQA): Slashing KV-Cache Memory
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
MQA shares a single Key/Value head across all Query heads; GQA groups Query heads to share Key/Value heads, slashing inference KV-cache by with zero quality degradation.
📌 Key Takeaways
- •Multi-Head Attention (MHA): query heads, key heads, value heads; full KV cache redundancy
- •Multi-Query Attention (MQA / Shazeer 2019): query heads, key head, value head; cuts KV cache by factor ( reduction)
- •Grouped-Query Attention (GQA / Ainslie et al. 2023): query heads grouped into groups, each group sharing key and value head ( standard in LLaMA-3)
📐 Mathematical Derivations
Mathematical Formulations and KV Cache Reductions:
Let model have hidden dimension , query head count , head dimension , and sequence length .
① KV Cache Memory Formula per Token:
(in FP16).
- MHA: . Total KV memory .
- MQA: . Total KV memory . Cuts KV memory and memory bandwidth by factor (e.g., reduction!).
- GQA: Partition query heads into groups (each group has query heads sharing 1 KV head). . Cuts KV memory by factor (e.g., reduction when ).
② Why GQA is the Frontier Standard (LLaMA-3, Mistral):
While MQA slashes memory drastically, it suffers from minor capacity drops in large models () and training instabilities. GQA () achieves virtually identical speedups to MQA while recovering of MHA's modeling perplexity and task reasoning accuracy.
Let model have hidden dimension , query head count , head dimension , and sequence length .
① KV Cache Memory Formula per Token:
(in FP16).
- MHA: . Total KV memory .
- MQA: . Total KV memory . Cuts KV memory and memory bandwidth by factor (e.g., reduction!).
- GQA: Partition query heads into groups (each group has query heads sharing 1 KV head). . Cuts KV memory by factor (e.g., reduction when ).
② Why GQA is the Frontier Standard (LLaMA-3, Mistral):
While MQA slashes memory drastically, it suffers from minor capacity drops in large models () and training instabilities. GQA () achieves virtually identical speedups to MQA while recovering of MHA's modeling perplexity and task reasoning accuracy.
🏭 Production Trade-offs
Serving throughput impact: During autoregressive LLM decoding, inference is strictly memory-bandwidth bound. Slashing KV cache size by with GQA directly enables larger serving batch sizes and up to higher generation throughput on GPUs.
⚠️ Common Interview Pitfalls
- ✕Confusing Query heads with Key/Value heads; GQA leaves the number of Query heads () completely untouched
- ✕Assuming GQA reduces computational FLOPs; FLOP reduction is minor (), the massive gain is in memory bandwidth
🎯 Interviewer Follow-up Inquiries
- ?Why is autoregressive LLM inference memory-bandwidth bound rather than compute bound?
- ?How does uptraining allow an existing MHA pretrained model to convert into a GQA model with minimal compute?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚