D3Medium★ Core Essential · Industrial BedrockPart D · Attention Mechanisms & Transformer Blocks

Grouped-Query & Multi-Query Attention

⚡Flashcard Drill➔

Share K/V heads across Query head groups (GQA/MQA) to cut autoregressive KV-cache footprint and memory bandwidth.

⏱️ Time: O(B * H_q * S^2 * D_k) 计算量微弱节省
💾 Space: KV Cache 显存开销降为原先的 H_kv / H_q
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def repeat_kv(x: np.ndarray, num_rep: int) -> np.ndarray:
"""
将 KV 头重复展开以对齐 Query 头的数量。
x 形状: (B, H_kv, S, D_k)
返回: (B, H_q, S, D_k),其中 H_q = H_kv * num_rep
"""
if num_rep == 1:
return x
B, H_kv, S, D_k = x.shape
# 增加扩展维度并广播
x =
x = np.repeat(x, num_rep, axis=2) # (B, H_kv, num_rep, S, D_k)
return x.reshape(B, H_kv * num_rep, S, D_k)
def grouped_query_attention(
q: np.ndarray, # (B, H_q, S, D_k)
k: np.ndarray, # (B, H_kv, S, D_k)
v: np.ndarray, # (B, H_kv, S, D_k)
is_causal: bool = True
) -> np.ndarray:
B, H_q, S, D_k = q.shape
H_kv = k.shape[1]
assert H_q % H_kv == 0, "Query 头数必须是 KV 头数的整数倍"
num_rep = H_q // H_kv
# 将 KV 广播扩展为与 Q 头数相同
k_expanded = repeat_kv(k, num_rep) # (B, H_q, S, D_k)
v_expanded = repeat_kv(v, num_rep) # (B, H_q, S, D_k)
# 执行常规多头注意力点积
scores = np.matmul(q, k_expanded.swapaxes(-1, -2)) / np.sqrt(D_k)
if is_causal:
mask = np.triu(np.ones((S, S), dtype=bool), k=1)
scores = np.where(mask, -1e9, scores)
scores_max = np.max(scores, axis=-1, keepdims=True)
probs = np.exp(scores - scores_max)
probs /= np.sum(probs, axis=-1, keepdims=True)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What further KV-cache innovation does DeepSeek-V2 / V3 MLA (Multi-Head Latent Attention) introduce over GQA?
Next: KV Cache Autoregressive Generation Loop →
💡

Core Mental Anchor / Mnemonic

Group query heads to share K/V heads, slashing KV-cache HBM reads cleanly.

📐 Mathematical Derivation & Core Formula

HQ 个 Query 头分为 G 组,每组共享 1 个 Key/Value 头,KV 显存缩减至 1HQ/HKVH_Q \text{ 个 Query 头分为 } G \text{ 组,每组共享 } 1 \text{ 个 Key/Value 头,KV 显存缩减至 } \frac{1}{H_Q / H_{KV}}
### KV Cache Memory-Bandwidth Bottleneck
Autoregressive LLM decoding is memory-bandwidth-bound: every generated token must load the entire cached Key and Value history from HBM into on-chip SRAM.
- MHA: HQ=HKVH_Q = H_{KV}; each query head has dedicated K/V heads (largest KV cache);
- MQA: HKV=1H_{KV} = 1; all query heads share one K/V head (smallest cache, slight quality drop);
- GQA: Divides HQH_Q query heads into GG groups sharing HKV=GH_{KV} = G heads (e.g., LLaMA-3 uses HQ=32,HKV=8H_Q=32, H_{KV}=8, cutting KV cache by 4×4\times while matching MHA quality closely).

🔄 Tensor Dimensions & Shape Flow

k: (B, H_kv, S, D_k) -> repeat_kv -> (B, H_q, S, D_k) -> dot with q: (B, H_q, S, D_k) -> out

🛡️ Numerical Stability & Pitfalls Checklist

  • Optimized CUDA/Triton kernels index K/V heads via modular arithmetic without copying repeated tensors.
  • Store only H_kv heads in KV cache, reducing cache memory to H_kv / H_q of standard MHA.

💻 Reference Code Implementation

import numpy as np

def repeat_kv(x: np.ndarray, num_rep: int) -> np.ndarray:
    """
    将 KV 头重复展开以对齐 Query 头的数量。
    x 形状: (B, H_kv, S, D_k)
    返回: (B, H_q, S, D_k),其中 H_q = H_kv * num_rep
    """
    if num_rep == 1:
        return x
    B, H_kv, S, D_k = x.shape
    # 增加扩展维度并广播
    x = x[:, :, np.newaxis, :, :]  # (B, H_kv, 1, S, D_k)
    x = np.repeat(x, num_rep, axis=2)  # (B, H_kv, num_rep, S, D_k)
    return x.reshape(B, H_kv * num_rep, S, D_k)

def grouped_query_attention(
    q: np.ndarray,      # (B, H_q, S, D_k)
    k: np.ndarray,      # (B, H_kv, S, D_k)
    v: np.ndarray,      # (B, H_kv, S, D_k)
    is_causal: bool = True
) -> np.ndarray:
    B, H_q, S, D_k = q.shape
    H_kv = k.shape[1]
    assert H_q % H_kv == 0, "Query 头数必须是 KV 头数的整数倍"
    num_rep = H_q // H_kv
    
    # 将 KV 广播扩展为与 Q 头数相同
    k_expanded = repeat_kv(k, num_rep)  # (B, H_q, S, D_k)
    v_expanded = repeat_kv(v, num_rep)  # (B, H_q, S, D_k)
    
    # 执行常规多头注意力点积
    scores = np.matmul(q, k_expanded.swapaxes(-1, -2)) / np.sqrt(D_k)
    if is_causal:
        mask = np.triu(np.ones((S, S), dtype=bool), k=1)
        scores = np.where(mask, -1e9, scores)
    
    scores_max = np.max(scores, axis=-1, keepdims=True)
    probs = np.exp(scores - scores_max)
    probs /= np.sum(probs, axis=-1, keepdims=True)
    return np.matmul(probs, v_expanded)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, S, D_k = 2, 4, 8
H_q = 8
H_kv = 2  # 4 个 Q 头共享 1 个 KV 头
q = np.random.randn(B, H_q, S, D_k)
k = np.random.randn(B, H_kv, S, D_k)
v = np.random.randn(B, H_kv, S, D_k)
out = grouped_query_attention(q, k, v)
assert out.shape == (B, H_q, S, D_k), "输出形状不正确"
print("✓ GQA 分组查询注意力自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What further KV-cache innovation does DeepSeek-V2 / V3 MLA (Multi-Head Latent Attention) introduce over GQA?
Rather than simply sharing K/V heads, MLA projects Keys and Values jointly into a compact low-rank latent vector cached in memory and uses decoupled RoPE, shrinking KV cache footprint dramatically relative to standard MHA.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Multi-Head Attention (MHA)Back to Part D & All 105Next: KV Cache Autoregressive Generation Loop →