Attention Variants MHA/MQA/GQA

MHA (Multi-Head Attention) gives every head its own K/V projections; MQA (Multi-Query Attention) shares a single K/V pair across all heads; GQA (Grouped-Query Attention) partitions the $H$ query heads into $G$ groups that share K/V — a middle ground: LLaMA-2/3 70B uses 8 KV heads, LLaMA-2 7B uses MQA, and Mistral/Gemma/Qwen all adopt GQA.

any autoregressive serving scenario where KV-cache memory is the constraint; since 2023 GQA is the de-facto standard in open models (LLaMA-2/3, Mistral, Gemma, Qwen).

at decode time every token caches all history K/V, and memory grows linearly with sequence length: cache size $= 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b$ (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); LLaMA-3 70B at $s = 8192$, $b = 32$ needs about 687GB with MHA — dwarfing its 140GB of weights — and about 86GB with GQA (8 groups), saving ~87%; MQA saves the most but degrades quality slightly, so it is mostly used in small models.

Test Your Knowledge: Practice Questions for "Attention Variants MHA/MQA/GQA"

Single choice pitfall questions with instant feedback and mistake tracking.