M4-043M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Medium
⚡Flashcard Drill➔Mastery:

Cross-Attention vs Self-Attention: Structural Differences and Multimodal Roles

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-044 →
📐 Mathematical Definition
self:Q,K,V=XWQKV;cross:Q=YWQ, K,V=XWKV\text{self}: Q,K,V=XW^{QKV};\qquad \text{cross}: Q=Y W^{Q},\ K,V=XW^{KV}
⚡ One-Sentence Core Answer
Self-attention derives Query, Key, and Value from the same sequence; Cross-attention derives Query from the target sequence and Key/Value from an external conditioning source.

📌 Key Takeaways

  • •
    Self-Attention: Q,K,V=XWQ,XWK,XWVQ, K, V = X W_Q, X W_K, X W_V; models intra-sequence dependencies; square N×NN \times N or causal triangular matrix
  • •
    Cross-Attention: Q=YWQ,  K=XWK,  V=XWVQ = Y W_Q, \; K = X W_K, \; V = X W_V; rectangular attention matrix NY×NXN_Y \times N_X; models conditional alignment
  • •
    Applications: Encoder-Decoder translation (T5), Diffusion conditioning (Stable Diffusion text-to-image), Multimodal perception (Flamingo)

📐 Mathematical Derivations

Structural Comparison:
Let source sequence be X∈RNx×dxX \in \mathbb{R}^{N_x \times d_x} and target sequence be Y∈RNy×dyY \in \mathbb{R}^{N_y \times d_y}.
① Self-Attention:
Q=XWQ,K=XWK,V=XWVQ = X W_Q, \quad K = X W_K, \quad V = X W_V.
Attention matrix: A=softmax(QKTd)∈RNx×NxA = \text{softmax}\left( \frac{Q K^T}{\sqrt{d}} \right) \in \mathbb{R}^{N_x \times N_x}.
Every token routes information to and from tokens within its own sequence. Computes contextual representations.
② Cross-Attention:
Target acts as Query; Source acts as Key and Value:
Q=YWQ∈RNy×d,K=XWK∈RNx×d,V=XWV∈RNx×dQ = Y W_Q \in \mathbb{R}^{N_y \times d}, \quad K = X W_K \in \mathbb{R}^{N_x \times d}, \quad V = X W_V \in \mathbb{R}^{N_x \times d}.
Attention matrix: A=softmax(QKTd)∈RNy×NxA = \text{softmax}\left( \frac{Q K^T}{\sqrt{d}} \right) \in \mathbb{R}^{N_y \times N_x}.
Output: O=AV∈RNy×dO = A V \in \mathbb{R}^{N_y \times d}.
Each target token dynamically queries the entire source memory. No causal mask is applied along NxN_x because the conditioning context is fully available.

🏭 Production Trade-offs

Serving optimization in Cross-Attention: Source sequence XX is processed only once. Its Key and Value tensors (KX,VXK_X, V_X) are precomputed and frozen, allowing the target decoder to autoregressively query static KV states with zero recomputation.
⚠️ Common Interview Pitfalls
  • ✕
    Applying causal masking along the source dimension in cross-attention; conditioning context is fully visible
  • ✕
    Recomputing cross-attention Key and Value tensors at every autoregressive decoding step instead of caching them
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Stable Diffusion use cross-attention to condition image latent denoising on text embeddings?
  • ?
    Why is the KV-cache of cross-attention static throughout autoregressive decoding while self-attention KV-cache grows dynamically?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-042: Native Sparse Attention (NSA): Multi-Branch Hardware-Aligned Sparse Design📋 Back to BankNext →M4-044: Attention Entropy Collapse and How QK-Norm Resolves It