M4-036M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Easy
⚡Flashcard Drill➔Mastery:

Multi-Head Latent Attention (MLA): DeepSeek's Low-Rank KV Cache Compression

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-037 →
📐 Mathematical Definition
cKV=WDKVh;kC=WUKcKV;vC=WUVcKV;cache cKV (low-dim)c^{KV}=W^{DKV}h;\quad k^{C}=W^{UK}c^{KV};\quad v^{C}=W^{UV}c^{KV};\qquad \text{cache }c^{KV}\ (\text{low-dim})
⚡ One-Sentence Core Answer
MLA projects Key and Value vectors into a single low-dimensional latent vector via low-rank compression; caching only the compressed vector slashes KV cache by over 90% while outperforming MHA.

📌 Key Takeaways

  • •
    Low-rank compression: projects hidden state hth_t to latent vector ctKV∈Rdcc_t^{\text{KV}} \in \mathbb{R}^{d_c} (dc≪nh⋅dkd_c \ll n_h \cdot d_k), caching only ctKVc_t^{\text{KV}}
  • •
    Matrix absorption: during inference, up-projection matrices are mathematically folded into Query projections, eliminating intermediate vector materialization
  • •
    Decoupled RoPE: separates positional rotary information into a tiny standalone vector (dR=64d_R = 64) to preserve low-rank absorption

📐 Mathematical Derivations

Mathematical Formulations (DeepSeek-V2 / DeepSeek-V3 Technical Reports, 2024):
Standard MHA caches 2⋅nh⋅dk2 \cdot n_h \cdot d_k elements per token. For nh=128,dk=128n_h=128, d_k=128, this is 32,76832,768 values.
① Low-Rank KV Compression:
Instead of projecting and caching full multi-head keys and values, compress hidden state ht∈Rdh_t \in \mathbb{R}^d into a compressed latent vector:
ctKV=WDKVht∈Rdcc_t^{\text{KV}} = W_{\text{DKV}} h_t \in \mathbb{R}^{d_c}, where dc=512d_c = 512.
Only ctKVc_t^{\text{KV}} is stored in the KV Cache! (Plus a tiny decoupled RoPE key ktR∈R64k_t^R \in \mathbb{R}^{64}).
Total cache per token: 512+64=576512 + 64 = 576 elements (a massive $93.3\%$ reduction compared to standard MHA).
② Matrix Absorption Trick in Inference:
In training, keys and values are up-projected: K=cKVWUKK = c^{\text{KV}} W_{\text{UK}}, V=cKVWUVV = c^{\text{KV}} W_{\text{UV}}.
In inference, we avoid uncompressing KK and VV by absorbing weights into Query and Output projections:
QKT=(hWQ)(cKVWUK)T=h(WQWUKT)(cKV)T=Q~(cKV)TQ K^T = (h W_Q) (c^{\text{KV}} W_{\text{UK}})^T = h (W_Q W_{\text{UK}}^T) (c^{\text{KV}})^T = \tilde{Q} (c^{\text{KV}})^T, where Q~=h(WQWUKT)\tilde{Q} = h (W_Q W_{\text{UK}}^T).
Similarly, AV=A(cKVWUV)=(AcKV)WUVA V = A (c^{\text{KV}} W_{\text{UV}}) = (A c^{\text{KV}}) W_{\text{UV}}.
Attention computes directly between absorbed queries and the compressed latent cache cKVc^{\text{KV}}!

🏭 Production Trade-offs

Architecture milestone: MLA is widely recognized as one of the most brilliant architectural innovations in modern LLM design, allowing DeepSeek-V2 and V3 to serve hundreds of concurrent users per GPU node with negligible memory pressure.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to absorb RoPE-transformed keys into the query projection; RoPE is non-linear with respect to token position, which is why MLA strictly decouples RoPE into a separate branch
  • ✕
    Assuming MLA degrades model capacity like MQA; MLA actually matches or exceeds MHA modeling perplexity
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Rotary Position Embedding (RoPE) prevent standard matrix absorption, and how does Decoupled RoPE solve it?
  • ?
    How does MLA compare to Grouped-Query Attention (GQA) in terms of KV-cache compression and serving throughput?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-035: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA): Slashing KV-Cache Memory📋 Back to BankNext →M4-037: Sparse Attention Mechanisms and Sliding Window Attention (SWA)