D2Medium★ Core Essential · Industrial BedrockPart D · Attention Mechanisms & Transformer Blocks

Multi-Head Attention (MHA)

⚡Flashcard Drill➔

Project Q, K, V into parallel attention heads across subspaces, concatenate heads, and project output.

⏱️ Time: O(B * S^2 * D + B * S * D^2)
💾 Space: O(B * H * S^2) 注意力矩阵缓存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int):
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k =
# 权重参数 (为简化展示采用随机初始化)
self.W_q = np.random.randn(d_model, d_model) * 0.02
self.W_k = np.random.randn(d_model, d_model) * 0.02
self.W_v = np.random.randn(d_model, d_model) * 0.02
self.W_o = np.random.randn(d_model, d_model) * 0.02
def forward(self, x: np.ndarray, is_causal: bool = True) -> np.ndarray:
"""
x: (B, S, D)
"""
B, S, D = x.shape
# 1. 线性投影: (B, S, D) @ (D, D) -> (B, S, D)
Q = x @ self.W_q
K = x @ self.W_k
V = x @ self.W_v
# 2. 分头与维度置换: (B, S, H, D_k) -> (B, H, S, D_k)
Q = Q.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
K = K.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
V = V.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
# 3. 批量多头缩放点积
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(self.d_k)
if is_causal:
mask = np.triu(np.ones((S, S), dtype=bool), k=1)
scores = np.where(mask, -1e9, scores)
scores_max = np.max(scores, axis=-1, keepdims=True)
exp_s = np.exp(scores - scores_max)
attn_probs = exp_s / np.sum(exp_s, axis=-1, keepdims=True)
context = np.matmul(attn_probs, V) # (B, H, S, D_k)
# 4. 拼接多头并输出投影
# 置换回 (B, S, H, D_k) 并平铺为 (B, S, D)
context = context.swapaxes(1, 2).reshape(B, S, D)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does sequence padding waste compute in MHA when batch sequence lengths vary widely, and how is it solved?
Next: Grouped-Query & Multi-Query Attention →
💡

Core Mental Anchor / Mnemonic

Project, split heads, attend in parallel subspaces, transpose back, concat, and project with W_o.

📐 Mathematical Derivation & Core Formula

MHA(Q,K,V)=Concat(head1,…,headh)WO\mathrm{MHA}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h) W^O
### Multi-Subspace Attention Projections
Single-head attention averages all relationships into one weighting map. Multi-Head Attention (MHA) projects inputs into hh independent subspaces of dimension dk=D/hd_k = D/h:
headi=Attention(QWiQ,KWiK,VWiV)\mathrm{head}_i = \mathrm{Attention}(Q W_i^Q, K W_i^K, V W_i^V)
Different heads can specialize in syntactic dependencies, coreference, or long-range patterns. Concatenating all heads and projecting via WO∈RD×DW^O \in \mathbb{R}^{D \times D} mixes information across subspaces.

🔄 Tensor Dimensions & Shape Flow

(B, S, D) -> Q,K,V -> (B, S, H, D_k) -> transpose -> (B, H, S, D_k) -> Attn -> reshape (B, S, D) -> W_o

🛡️ Numerical Stability & Pitfalls Checklist

  • Maintain contiguous memory layout (.contiguous() in PyTorch) before reshaping concatenated heads.
  • Production layers often fuse W_q, W_k, W_v into one (D, 3*D) linear projection for fewer GEMM calls.

💻 Reference Code Implementation

import numpy as np

class MultiHeadAttention:
    def __init__(self, d_model: int, num_heads: int):
        assert d_model % num_heads == 0
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        # 权重参数 (为简化展示采用随机初始化)
        self.W_q = np.random.randn(d_model, d_model) * 0.02
        self.W_k = np.random.randn(d_model, d_model) * 0.02
        self.W_v = np.random.randn(d_model, d_model) * 0.02
        self.W_o = np.random.randn(d_model, d_model) * 0.02
        
    def forward(self, x: np.ndarray, is_causal: bool = True) -> np.ndarray:
        """
        x: (B, S, D)
        """
        B, S, D = x.shape
        # 1. 线性投影: (B, S, D) @ (D, D) -> (B, S, D)
        Q = x @ self.W_q
        K = x @ self.W_k
        V = x @ self.W_v
        
        # 2. 分头与维度置换: (B, S, H, D_k) -> (B, H, S, D_k)
        Q = Q.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
        K = K.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
        V = V.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
        
        # 3. 批量多头缩放点积
        scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(self.d_k)
        if is_causal:
            mask = np.triu(np.ones((S, S), dtype=bool), k=1)
            scores = np.where(mask, -1e9, scores)
        
        scores_max = np.max(scores, axis=-1, keepdims=True)
        exp_s = np.exp(scores - scores_max)
        attn_probs = exp_s / np.sum(exp_s, axis=-1, keepdims=True)
        context = np.matmul(attn_probs, V)  # (B, H, S, D_k)
        
        # 4. 拼接多头并输出投影
        # 置换回 (B, S, H, D_k) 并平铺为 (B, S, D)
        context = context.swapaxes(1, 2).reshape(B, S, D)
        return context @ self.W_o

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
mha = MultiHeadAttention(d_model=16, num_heads=4)
x = np.random.randn(2, 6, 16)
out = mha.forward(x, is_causal=True)
assert out.shape == (2, 6, 16), "输出形状错误"
assert not np.isnan(out).any(), "包含 NaN"
print("✓ MultiHeadAttention 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does sequence padding waste compute in MHA when batch sequence lengths vary widely, and how is it solved?
Padding short sequences up to the longest sample (e.g., 16 vs. 2048) wastes compute and HBM bandwidth on masked zeros. Production engines pack variable-length tokens into a single 1D buffer using cumulative length offsets (cu_seqlens, e.g., FlashAttention varlen), eliminating padding overhead.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Scaled Dot-Product Attention with Causal MaskBack to Part D & All 105Next: Grouped-Query & Multi-Query Attention →