D2Medium★ Core Essential · Industrial BedrockPart D · Attention Mechanisms & Transformer Blocks
Multi-Head Attention (MHA)
Project Q, K, V into parallel attention heads across subspaces, concatenate heads, and project output.
⏱️ Time:
O(B * S^2 * D + B * S * D^2)💾 Space:
O(B * H * S^2) 注意力矩阵缓存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int):
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k =
# 权重参数 (为简化展示采用随机初始化)
self.W_q = np.random.randn(d_model, d_model) * 0.02
self.W_k = np.random.randn(d_model, d_model) * 0.02
self.W_v = np.random.randn(d_model, d_model) * 0.02
self.W_o = np.random.randn(d_model, d_model) * 0.02
def forward(self, x: np.ndarray, is_causal: bool = True) -> np.ndarray:
"""
x: (B, S, D)
"""
B, S, D = x.shape
# 1. 线性投影: (B, S, D) @ (D, D) -> (B, S, D)
Q = x @ self.W_q
K = x @ self.W_k
V = x @ self.W_v
# 2. 分头与维度置换: (B, S, H, D_k) -> (B, H, S, D_k)
Q = Q.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
K = K.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
V = V.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
# 3. 批量多头缩放点积
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(self.d_k)
if is_causal:
mask = np.triu(np.ones((S, S), dtype=bool), k=1)
scores = np.where(mask, -1e9, scores)
scores_max = np.max(scores, axis=-1, keepdims=True)
exp_s = np.exp(scores - scores_max)
attn_probs = exp_s / np.sum(exp_s, axis=-1, keepdims=True)
context = np.matmul(attn_probs, V) # (B, H, S, D_k)
# 4. 拼接多头并输出投影
# 置换回 (B, S, H, D_k) 并平铺为 (B, S, D)
context = context.swapaxes(1, 2).reshape(B, S, D)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does sequence padding waste compute in MHA when batch sequence lengths vary widely, and how is it solved?
💡
Core Mental Anchor / Mnemonic
Project, split heads, attend in parallel subspaces, transpose back, concat, and project with W_o.
📐 Mathematical Derivation & Core Formula
### Multi-Subspace Attention Projections
Single-head attention averages all relationships into one weighting map. Multi-Head Attention (MHA) projects inputs into independent subspaces of dimension :
Different heads can specialize in syntactic dependencies, coreference, or long-range patterns. Concatenating all heads and projecting via mixes information across subspaces.
Single-head attention averages all relationships into one weighting map. Multi-Head Attention (MHA) projects inputs into independent subspaces of dimension :
Different heads can specialize in syntactic dependencies, coreference, or long-range patterns. Concatenating all heads and projecting via mixes information across subspaces.
🔄 Tensor Dimensions & Shape Flow
(B, S, D) -> Q,K,V -> (B, S, H, D_k) -> transpose -> (B, H, S, D_k) -> Attn -> reshape (B, S, D) -> W_o
🛡️ Numerical Stability & Pitfalls Checklist
- Maintain contiguous memory layout (
.contiguous()in PyTorch) before reshaping concatenated heads. - Production layers often fuse W_q, W_k, W_v into one (D, 3*D) linear projection for fewer GEMM calls.
💻 Reference Code Implementation
import numpy as np
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int):
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
# 权重参数 (为简化展示采用随机初始化)
self.W_q = np.random.randn(d_model, d_model) * 0.02
self.W_k = np.random.randn(d_model, d_model) * 0.02
self.W_v = np.random.randn(d_model, d_model) * 0.02
self.W_o = np.random.randn(d_model, d_model) * 0.02
def forward(self, x: np.ndarray, is_causal: bool = True) -> np.ndarray:
"""
x: (B, S, D)
"""
B, S, D = x.shape
# 1. 线性投影: (B, S, D) @ (D, D) -> (B, S, D)
Q = x @ self.W_q
K = x @ self.W_k
V = x @ self.W_v
# 2. 分头与维度置换: (B, S, H, D_k) -> (B, H, S, D_k)
Q = Q.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
K = K.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
V = V.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
# 3. 批量多头缩放点积
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(self.d_k)
if is_causal:
mask = np.triu(np.ones((S, S), dtype=bool), k=1)
scores = np.where(mask, -1e9, scores)
scores_max = np.max(scores, axis=-1, keepdims=True)
exp_s = np.exp(scores - scores_max)
attn_probs = exp_s / np.sum(exp_s, axis=-1, keepdims=True)
context = np.matmul(attn_probs, V) # (B, H, S, D_k)
# 4. 拼接多头并输出投影
# 置换回 (B, S, H, D_k) 并平铺为 (B, S, D)
context = context.swapaxes(1, 2).reshape(B, S, D)
return context @ self.W_o
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
mha = MultiHeadAttention(d_model=16, num_heads=4)
x = np.random.randn(2, 6, 16)
out = mha.forward(x, is_causal=True)
assert out.shape == (2, 6, 16), "输出形状错误"
assert not np.isnan(out).any(), "包含 NaN"
print("✓ MultiHeadAttention 自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does sequence padding waste compute in MHA when batch sequence lengths vary widely, and how is it solved?
Padding short sequences up to the longest sample (e.g., 16 vs. 2048) wastes compute and HBM bandwidth on masked zeros. Production engines pack variable-length tokens into a single 1D buffer using cumulative length offsets (
cu_seqlens, e.g., FlashAttention varlen), eliminating padding overhead.⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.