M4-035M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Easy
⚡Flashcard Drill➔Mastery:

Multi-Query Attention (MQA) and Grouped-Query Attention (GQA): Slashing KV-Cache Memory

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-036 →
📐 Mathematical Definition
MHA:h KV heads;GQA:g KV heads;MQA:1 KV head;KV∝nkv\text{MHA}: h\ \text{KV heads};\quad \text{GQA}: g\ \text{KV heads};\quad \text{MQA}: 1\ \text{KV head};\qquad \text{KV}\propto n_{\text{kv}}
⚡ One-Sentence Core Answer
MQA shares a single Key/Value head across all Query heads; GQA groups Query heads to share Key/Value heads, slashing inference KV-cache by 4−8×4-8\times with zero quality degradation.

📌 Key Takeaways

  • •
    Multi-Head Attention (MHA): HH query heads, HH key heads, HH value heads; full KV cache redundancy
  • •
    Multi-Query Attention (MQA / Shazeer 2019): HH query heads, 11 key head, 11 value head; cuts KV cache by factor HH (97%97\% reduction)
  • •
    Grouped-Query Attention (GQA / Ainslie et al. 2023): HH query heads grouped into GG groups, each group sharing 11 key and 11 value head (G=8G=8 standard in LLaMA-3)

📐 Mathematical Derivations

Mathematical Formulations and KV Cache Reductions:
Let model have hidden dimension dd, query head count HH, head dimension dk=d/Hd_k = d/H, and sequence length SS.
① KV Cache Memory Formula per Token:
Memorytoken=2×[2⋅L⋅nkv⋅dk⋅B] bytes\text{Memory}_{\text{token}} = 2 \times [2 \cdot L \cdot n_{\text{kv}} \cdot d_k \cdot B] \text{ bytes} (in FP16).
- MHA: nkv=Hn_{\text{kv}} = H. Total KV memory ∝H⋅dk=d\propto H \cdot d_k = d.
- MQA: nkv=1n_{\text{kv}} = 1. Total KV memory ∝1⋅dk=d/H\propto 1 \cdot d_k = d / H. Cuts KV memory and memory bandwidth by factor HH (e.g., 32×32\times reduction!).
- GQA: Partition HH query heads into GG groups (each group has H/GH/G query heads sharing 1 KV head). nkv=Gn_{\text{kv}} = G. Cuts KV memory by factor H/GH/G (e.g., 8×8\times reduction when H=64,G=8H=64, G=8).
② Why GQA is the Frontier Standard (LLaMA-3, Mistral):
While MQA slashes memory drastically, it suffers from minor capacity drops in large models (>30B>30\text{B}) and training instabilities. GQA (G=8G=8) achieves virtually identical speedups to MQA while recovering 100%100\% of MHA's modeling perplexity and task reasoning accuracy.

🏭 Production Trade-offs

Serving throughput impact: During autoregressive LLM decoding, inference is strictly memory-bandwidth bound. Slashing KV cache size by 8×8\times with GQA directly enables 8×8\times larger serving batch sizes and up to 4×4\times higher generation throughput on GPUs.
⚠️ Common Interview Pitfalls
  • ✕
    Confusing Query heads with Key/Value heads; GQA leaves the number of Query heads (HH) completely untouched
  • ✕
    Assuming GQA reduces computational FLOPs; FLOP reduction is minor (∼5%\sim 5\%), the massive gain is in memory bandwidth
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is autoregressive LLM inference memory-bandwidth bound rather than compute bound?
  • ?
    How does uptraining allow an existing MHA pretrained model to convert into a GQA model with minimal compute?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-034: Benchmarking Positional Encodings and Context Length Extrapolation📋 Back to BankNext →M4-036: Multi-Head Latent Attention (MLA): DeepSeek's Low-Rank KV Cache Compression