M4-036M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Easy
⚡Flashcard Drill➔Mastery:
Multi-Head Latent Attention (MLA): DeepSeek's Low-Rank KV Cache Compression
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
MLA projects Key and Value vectors into a single low-dimensional latent vector via low-rank compression; caching only the compressed vector slashes KV cache by over 90% while outperforming MHA.
📌 Key Takeaways
- •Low-rank compression: projects hidden state to latent vector (), caching only
- •Matrix absorption: during inference, up-projection matrices are mathematically folded into Query projections, eliminating intermediate vector materialization
- •Decoupled RoPE: separates positional rotary information into a tiny standalone vector () to preserve low-rank absorption
📐 Mathematical Derivations
Mathematical Formulations (DeepSeek-V2 / DeepSeek-V3 Technical Reports, 2024):
Standard MHA caches elements per token. For , this is values.
① Low-Rank KV Compression:
Instead of projecting and caching full multi-head keys and values, compress hidden state into a compressed latent vector:
, where .
Only is stored in the KV Cache! (Plus a tiny decoupled RoPE key ).
Total cache per token: elements (a massive $93.3\%$ reduction compared to standard MHA).
② Matrix Absorption Trick in Inference:
In training, keys and values are up-projected: , .
In inference, we avoid uncompressing and by absorbing weights into Query and Output projections:
, where .
Similarly, .
Attention computes directly between absorbed queries and the compressed latent cache !
Standard MHA caches elements per token. For , this is values.
① Low-Rank KV Compression:
Instead of projecting and caching full multi-head keys and values, compress hidden state into a compressed latent vector:
, where .
Only is stored in the KV Cache! (Plus a tiny decoupled RoPE key ).
Total cache per token: elements (a massive $93.3\%$ reduction compared to standard MHA).
② Matrix Absorption Trick in Inference:
In training, keys and values are up-projected: , .
In inference, we avoid uncompressing and by absorbing weights into Query and Output projections:
, where .
Similarly, .
Attention computes directly between absorbed queries and the compressed latent cache !
🏭 Production Trade-offs
Architecture milestone: MLA is widely recognized as one of the most brilliant architectural innovations in modern LLM design, allowing DeepSeek-V2 and V3 to serve hundreds of concurrent users per GPU node with negligible memory pressure.
⚠️ Common Interview Pitfalls
- ✕Attempting to absorb RoPE-transformed keys into the query projection; RoPE is non-linear with respect to token position, which is why MLA strictly decouples RoPE into a separate branch
- ✕Assuming MLA degrades model capacity like MQA; MLA actually matches or exceeds MHA modeling perplexity
🎯 Interviewer Follow-up Inquiries
- ?Why does Rotary Position Embedding (RoPE) prevent standard matrix absorption, and how does Decoupled RoPE solve it?
- ?How does MLA compare to Grouped-Query Attention (GQA) in terms of KV-cache compression and serving throughput?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚