M4-043M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Medium
⚡Flashcard Drill➔Mastery:
Cross-Attention vs Self-Attention: Structural Differences and Multimodal Roles
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Self-attention derives Query, Key, and Value from the same sequence; Cross-attention derives Query from the target sequence and Key/Value from an external conditioning source.
📌 Key Takeaways
- •Self-Attention: ; models intra-sequence dependencies; square or causal triangular matrix
- •Cross-Attention: ; rectangular attention matrix ; models conditional alignment
- •Applications: Encoder-Decoder translation (T5), Diffusion conditioning (Stable Diffusion text-to-image), Multimodal perception (Flamingo)
📐 Mathematical Derivations
Structural Comparison:
Let source sequence be and target sequence be .
① Self-Attention:
.
Attention matrix: .
Every token routes information to and from tokens within its own sequence. Computes contextual representations.
② Cross-Attention:
Target acts as Query; Source acts as Key and Value:
.
Attention matrix: .
Output: .
Each target token dynamically queries the entire source memory. No causal mask is applied along because the conditioning context is fully available.
Let source sequence be and target sequence be .
① Self-Attention:
.
Attention matrix: .
Every token routes information to and from tokens within its own sequence. Computes contextual representations.
② Cross-Attention:
Target acts as Query; Source acts as Key and Value:
.
Attention matrix: .
Output: .
Each target token dynamically queries the entire source memory. No causal mask is applied along because the conditioning context is fully available.
🏭 Production Trade-offs
Serving optimization in Cross-Attention: Source sequence is processed only once. Its Key and Value tensors () are precomputed and frozen, allowing the target decoder to autoregressively query static KV states with zero recomputation.
⚠️ Common Interview Pitfalls
- ✕Applying causal masking along the source dimension in cross-attention; conditioning context is fully visible
- ✕Recomputing cross-attention Key and Value tensors at every autoregressive decoding step instead of caching them
🎯 Interviewer Follow-up Inquiries
- ?How does Stable Diffusion use cross-attention to condition image latent denoising on text embeddings?
- ?Why is the KV-cache of cross-attention static throughout autoregressive decoding while self-attention KV-cache grows dynamically?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚