M4-016M4: Sequences & TransformersTransformer Architecture AnatomyMedium
⚡Flashcard Drill➔Mastery:

Why Multi-Head Attention (MHA) is Effective: Subspace Projections and Diversity

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-017 →
📐 Mathematical Definition
MHA(X)=Concat(head1,…,headh)WO,headi=Attn(XWiQ,XWiK,XWiV)\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)W^O,\quad \mathrm{head}_i=\mathrm{Attn}(XW_i^Q,XW_i^K,XW_i^V)
⚡ One-Sentence Core Answer
Multi-Head Attention projects representations into multiple lower-dimensional subspaces simultaneously, allowing the model to attend to diverse syntactic, semantic, and positional relations concurrently.

📌 Key Takeaways

  • •
    Subspace diversity: single-head attention averages all positions into one vector; multi-head allows head AA to track syntax while head BB tracks pronoun coreference
  • •
    FLOP invariance: splitting dd into hh heads of dimension dk=d/hd_k = d/h preserves exact total FLOPs and parameter counts (4d24 d^2)
  • •
    Ensemble perspective: acts as an ensemble of hh distinct attention heads operating in parallel

📐 Mathematical Derivations

Mathematical Formulation (Vaswani et al., 2017):
MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W_O,
where headi=Attention(QWiQ,KWiK,VWiV)\text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V), with WiQ,WiK,WiV∈Rd×dkW_i^Q, W_i^K, W_i^V \in \mathbb{R}^{d \times d_k} and WO∈Rhdk×dW_O \in \mathbb{R}^{h d_k \times d}.
- Parameter Preservation:
Total parameters across all hh heads:
h×[3×(d×dk)]+(hdk×d)=3d(hdk)+(hdk)d=3d2+d2=4d2h \times [ 3 \times (d \times d_k) ] + (h d_k \times d) = 3 d (h d_k) + (h d_k) d = 3 d^2 + d^2 = 4 d^2.
Because h×dk=dh \times d_k = d, computational FLOPs and parameter memory match a single full-dimensional attention head (dk=dd_k = d) exactly.
- Why Multiple Heads are Mathematically Superior:
In single-head attention, if word xtx_t must attend strongly to a preceding subject for verb agreement, its attention distribution places mass on the subject. It cannot simultaneously attend to an object or preposition without washing out the subject's signal. Multi-head attention allows distinct heads to attend to completely different token positions simultaneously in orthogonal subspaces.

🏭 Production Trade-offs

Head Count Trade-offs: Typical head dimension dkd_k is fixed to 64 or 128. Setting hh too large (dk<16d_k < 16) reduces subspace representation capacity; setting hh too small (h=1h=1) eliminates multi-aspect relational modeling.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming multi-head attention increases computational FLOPs compared to single-head attention; total FLOPs are identical
  • ✕
    Using different projection dimensions across heads, breaking parallel GPU batched matrix multiplication (bmm)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Multi-Head Attention incur identical FLOPs to single-head attention of equivalent hidden dimension?
  • ?
    What linguistic roles do individual attention heads specialize in (e.g., induction heads, previous token heads)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-015: Role of the Feed-Forward Network (FFN) and the 4x / (8/3)x Hidden Dimension Rationale📋 Back to BankNext →M4-017: Theoretical Expressive Power and Computational Limits of Transformers