Attention Variants MHA/MQA/GQA

MHA (Multi-Head Attention) gives every head its own K/V projections; MQA (Multi-Query Attention) shares a single K/V pair across all heads; GQA (Grouped-Query Attention) partitions the $H$ query heads into $G$ groups that share K/V — a middle ground: LLaMA-2/3 70B uses 8 KV heads, LLaMA-2 7B uses MQA, and Mistral/Gemma/Qwen all adopt GQA.

any autoregressive serving scenario where KV-cache memory is the constraint; since 2023 GQA is the de-facto standard in open models (LLaMA-2/3, Mistral, Gemma, Qwen).

at decode time every token caches all history K/V, and memory grows linearly with sequence length: cache size $= 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b$ (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); LLaMA-3 70B at $s = 8192$, $b = 32$ needs about 687GB with MHA — dwarfing its 140GB of weights — and about 86GB with GQA (8 groups), saving ~87%; MQA saves the most but degrades quality slightly, so it is mostly used in small models.

What does GQA save vs MHA? Meaning of each term in the KV-cache formula $2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b$?

How to choose the number of groups G in GQA? The quality vs memory trade-off?

Difference between MQA and GQA? Pros and cons of each?

Why does sharing K/V barely affect training but matter a lot for inference (KV cache exists only at inference)?

Configs of mainstream models: KV-head counts for LLaMA-2/3, Mistral, Qwen?