D5EasyPart D · Attention Mechanisms & Transformer Blocks

Cross-Attention Mechanism

⚡Flashcard Drill➔

Attend target Queries against external encoder/conditioning Keys and Values across heterogeneous sequence lengths.

⏱️ Time: O(B * S_q * S_kv * D)
💾 Space: O(B * H * S_q * S_kv)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def cross_attention(
x_query: np.ndarray, # (B, S_q, D) 解码器序列
x_context: np.ndarray, # (B, S_kv, D) 编码器/文本提示序列
W_q: np.ndarray, # (D, D)
W_k: np.ndarray, # (D, D)
W_v: np.ndarray, # (D, D)
W_o: np.ndarray, # (D, D)
num_heads: int = 4
) -> np.ndarray:
B, S_q, D = x_query.shape
S_kv = x_context.shape[1]
d_k =
# 1. 分别从不同来源投影 Q 与 K, V
Q = (x_query @ W_q).reshape(B, S_q, num_heads, d_k).swapaxes(1, 2)
K = (x_context @ W_k).reshape(B, S_kv, num_heads, d_k).swapaxes(1, 2)
V = (x_context @ W_v).reshape(B, S_kv, num_heads, d_k).swapaxes(1, 2)
# 2. 点积相关度打分: (B, H, S_q, d_k) @ (B, H, d_k, S_kv) -> (B, H, S_q, S_kv)
scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(d_k)
# Cross-Attention 通常不使用因果掩码 (解码词可以看到编码端所有上下文)
scores_max = np.max(scores, axis=-1, keepdims=True)
attn = np.exp(scores - scores_max)
attn = attn / np.sum(attn, axis=-1, keepdims=True)
# 3. 聚合编码端信息
out = np.matmul(attn, V) # (B, H, S_q, d_k)
out = out.swapaxes(1, 2).reshape(B, S_q, D)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: In Encoder-Decoder inference, how does Cross-Attention KV caching differ from Self-Attention?
Next: FlashAttention Online Softmax Block-wise Algorithm →
💡

Core Mental Anchor / Mnemonic

Queries come from decoder/target; Keys and Values come from external encoder or conditioning tokens.

📐 Mathematical Derivation & Core Formula

Q=XdecWQ,K=XencWK,V=XencWVQ = X_{\text{dec}} W_Q, \quad K = X_{\text{enc}} W_K, \quad V = X_{\text{enc}} W_V
### Cross-Sequence Conditioning Alignment
Unlike Self-Attention (Q,K,VQ, K, V from one sequence XX), Cross-Attention connects two distinct sequences:
- Queries ($Q$): Come from the decoder/target sequence of shape (B,Sq,D)(B, S_q, D);
- Keys & Values ($K, V$): Come from external context (e.g., encoder output or text prompt embeddings) of shape (B,Skv,D)(B, S_{kv}, D).
Computing QK⊤Q K^\top lets each target token attend selectively over relevant source/conditioning positions.

🔄 Tensor Dimensions & Shape Flow

x_q: (B, Sq, D), x_ctx: (B, Skv, D) -> Q (Sq), K/V (Skv) -> logits: (B, H, Sq, Skv) -> @ V -> (B, Sq, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Query length S_q and context length S_kv can differ; scores have shape (S_q, S_kv).
  • Apply padding masks over context_mask when encoder/prompt sequences have variable lengths.
  • Core mechanism injecting text prompt embeddings into U-Net spatial latents in Stable Diffusion.

💻 Reference Code Implementation

import numpy as np

def cross_attention(
    x_query: np.ndarray,     # (B, S_q, D) 解码器序列
    x_context: np.ndarray,   # (B, S_kv, D) 编码器/文本提示序列
    W_q: np.ndarray,         # (D, D)
    W_k: np.ndarray,         # (D, D)
    W_v: np.ndarray,         # (D, D)
    W_o: np.ndarray,         # (D, D)
    num_heads: int = 4
) -> np.ndarray:
    B, S_q, D = x_query.shape
    S_kv = x_context.shape[1]
    d_k = D // num_heads
    
    # 1. 分别从不同来源投影 Q 与 K, V
    Q = (x_query @ W_q).reshape(B, S_q, num_heads, d_k).swapaxes(1, 2)
    K = (x_context @ W_k).reshape(B, S_kv, num_heads, d_k).swapaxes(1, 2)
    V = (x_context @ W_v).reshape(B, S_kv, num_heads, d_k).swapaxes(1, 2)
    
    # 2. 点积相关度打分: (B, H, S_q, d_k) @ (B, H, d_k, S_kv) -> (B, H, S_q, S_kv)
    scores = np.matmul(Q, K.swapaxes(-1, -2)) / np.sqrt(d_k)
    
    # Cross-Attention 通常不使用因果掩码 (解码词可以看到编码端所有上下文)
    scores_max = np.max(scores, axis=-1, keepdims=True)
    attn = np.exp(scores - scores_max)
    attn = attn / np.sum(attn, axis=-1, keepdims=True)
    
    # 3. 聚合编码端信息
    out = np.matmul(attn, V)  # (B, H, S_q, d_k)
    out = out.swapaxes(1, 2).reshape(B, S_q, D)
    return out @ W_o

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, Sq, Skv, D = 2, 5, 12, 16
x_q = np.random.randn(B, Sq, D)
x_ctx = np.random.randn(B, Skv, D)
W = np.random.randn(D, D) * 0.02
out = cross_attention(x_q, x_ctx, W, W, W, W, num_heads=4)
assert out.shape == (B, Sq, D), f"输出形状应匹配 Query 长度: {out.shape}"
print("✓ Cross-Attention 跨模态注意力自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:In Encoder-Decoder inference, how does Cross-Attention KV caching differ from Self-Attention?
Encoder outputs stay constant throughout decoding, so Cross-Attention Keys and Values are computed once at the end of encoding/prefill and reused statically, whereas Decoder Masked Self-Attention appends one new K/V slice every decoding step.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: KV Cache Autoregressive Generation LoopBack to Part D & All 105Next: FlashAttention Online Softmax Block-wise Algorithm →