D2Medium★ Core Essential · Industrial BedrockPart D · Attention Mechanisms & Transformer Blocks
Multi-Head Attention (MHA)
Industrial-grade implementation and mathematical foundations of Multi-Head Attention (MHA).
⏱️ Time Complexity:
O(B * S^2 * D + B * S * D^2)💾 Space Complexity:
O(B * H * S^2) 注意力矩阵缓存💡
Core Mental Anchor / Mnemonic
Master Multi-Head Attention (MHA): enforce numerical stability, check tensor shapes, and eliminate redundant memory allocations.
📐 Mathematical Derivation & Core Formula
### Mathematical Derivation & Theoretical Principles
Detailed first-principles formulation and architectural mechanics for Multi-Head Attention (MHA).
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
Detailed first-principles formulation and architectural mechanics for Multi-Head Attention (MHA).
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
🔄 Tensor Dimensions & Shape Flow
(B, S, D) -> 投影为 Q,K,V -> (B, S, H, D_k) -> swapaxes -> (B, H, S, D_k) -> Attention -> (B, H, S, D_k) -> swapaxes & reshape -> (B, S, D) -> W_o 投影 -> (B, S, D)
🛡️ Industrial Numerical Stability & Pitfalls
- Ensure proper multi-dimensional tensor broadcasting and keepdims retention.
- Enforce numerical guards (eps clamping and overflow thresholds) during exponentiation and division.
- Verify train versus eval mode behavioral distinctions (e.g. frozen running statistics and dropout bypass).
💻 Industrial Code Implementation
import numpy as np
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int):
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
# 权重参数 (为简化展示采用随机初始化)
self.W_q = np.random.randn(d_model, d_model) * 0.02
self.W_k = np.random.randn(d_model, d_model) * 0.02
self.W_v = np.random.randn(d_model, d_model) * 0.02
self.W_o = np.random.randn(d_model, d_model) * 0.02
def forward(self, x: np.ndarray, is_causal: bool = True) -> np.ndarray:
"""
x: (B, S, D)
"""
B, S, D = x.shape
# 1. 线性投影: (B, S, D) @ (D, D) -> (B, S, D)
Q = x @ self.W_q
K = x @ self.W_k
V = x @ self.W_v
# 2. 分头与维度置换: (B, S, H, D_k) -> (B, H, S, D_k)
Q = Q.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
K = K.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
V = V.reshape(B, S, self.num_heads, self.d_k).swapaxes(1, 2)
# 3. 批量多头缩放点积
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(self.d_k)
if is_causal:
mask = np.triu(np.ones((S, S), dtype=bool), k=1)
scores = np.where(mask, -1e9, scores)
scores_max = np.max(scores, axis=-1, keepdims=True)
exp_s = np.exp(scores - scores_max)
attn_probs = exp_s / np.sum(exp_s, axis=-1, keepdims=True)
context = np.matmul(attn_probs, V) # (B, H, S, D_k)
# 4. 拼接多头并输出投影
# 置换回 (B, S, H, D_k) 并平铺为 (B, S, D)
context = context.swapaxes(1, 2).reshape(B, S, D)
return context @ self.W_o
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter to verify correctness:
import numpy as np
mha = MultiHeadAttention(d_model=16, num_heads=4)
x = np.random.randn(2, 6, 16)
out = mha.forward(x, is_causal=True)
assert out.shape == (2, 6, 16), "输出形状错误"
assert not np.isnan(out).any(), "包含 NaN"
print("✓ MultiHeadAttention 自测通过")🎯 Core Architecture Follow-up Q&A
Q1:What are the key trade-offs and memory bottlenecks when deploying Multi-Head Attention (MHA) in high-throughput inference?
Memory bandwidth (HBM to SRAM I/O) is the primary latency factor. Fusing element-wise operations and avoiding intermediate tensor materialization significantly outperforms naive implementations.
Q2:How does Multi-Head Attention (MHA) handle extreme numerical boundaries or precision reduction (FP16/BF16/INT8)?
Under low precision, operations must be upcasted to FP32 during accumulation to prevent underflow/overflow, followed by proper scaling and clamping before converting back to the target format.