M4-016M4: Sequences & TransformersTransformer Architecture AnatomyMedium
⚡Flashcard Drill➔Mastery:
Why Multi-Head Attention (MHA) is Effective: Subspace Projections and Diversity
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Multi-Head Attention projects representations into multiple lower-dimensional subspaces simultaneously, allowing the model to attend to diverse syntactic, semantic, and positional relations concurrently.
📌 Key Takeaways
- •Subspace diversity: single-head attention averages all positions into one vector; multi-head allows head to track syntax while head tracks pronoun coreference
- •FLOP invariance: splitting into heads of dimension preserves exact total FLOPs and parameter counts ()
- •Ensemble perspective: acts as an ensemble of distinct attention heads operating in parallel
📐 Mathematical Derivations
Mathematical Formulation (Vaswani et al., 2017):
,
where , with and .
- Parameter Preservation:
Total parameters across all heads:
.
Because , computational FLOPs and parameter memory match a single full-dimensional attention head () exactly.
- Why Multiple Heads are Mathematically Superior:
In single-head attention, if word must attend strongly to a preceding subject for verb agreement, its attention distribution places mass on the subject. It cannot simultaneously attend to an object or preposition without washing out the subject's signal. Multi-head attention allows distinct heads to attend to completely different token positions simultaneously in orthogonal subspaces.
,
where , with and .
- Parameter Preservation:
Total parameters across all heads:
.
Because , computational FLOPs and parameter memory match a single full-dimensional attention head () exactly.
- Why Multiple Heads are Mathematically Superior:
In single-head attention, if word must attend strongly to a preceding subject for verb agreement, its attention distribution places mass on the subject. It cannot simultaneously attend to an object or preposition without washing out the subject's signal. Multi-head attention allows distinct heads to attend to completely different token positions simultaneously in orthogonal subspaces.
🏭 Production Trade-offs
Head Count Trade-offs: Typical head dimension is fixed to 64 or 128. Setting too large () reduces subspace representation capacity; setting too small () eliminates multi-aspect relational modeling.
⚠️ Common Interview Pitfalls
- ✕Assuming multi-head attention increases computational FLOPs compared to single-head attention; total FLOPs are identical
- ✕Using different projection dimensions across heads, breaking parallel GPU batched matrix multiplication (
bmm)
🎯 Interviewer Follow-up Inquiries
- ?Why does Multi-Head Attention incur identical FLOPs to single-head attention of equivalent hidden dimension?
- ?What linguistic roles do individual attention heads specialize in (e.g., induction heads, previous token heads)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚