⚡ LLMsID: mla-attention

Multi-head Latent Attention

Q: How does MLA compress KV? Dimensionality of $c_t^{KV}$ and the reconstruction process?

【Multi-head Latent Attention 核心考点 1】How does MLA compress KV? Dimensionality of $c_t^{KV}$ and the reconstruction process?。技术解析：Definition: MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector c_t^{KV} \in \mathbb{R}^{d_c} (d_c = 512 in DeepSeek-V2), and per-head K/V are reconstructed by up-pro... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/mla-attention

Q: Why does MLA need decoupled RoPE? What role does $k_t^R$ play?

【Multi-head Latent Attention 核心考点 2】Why does MLA need decoupled RoPE? What role does $k_t^R$ play?。技术解析：Definition: MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector c_t^{KV} \in \mathbb{R}^{d_c} (d_c = 512 in DeepSeek-V2), and per-head K/V are reconstructed by up-pro... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/mla-attention

Q: How much memory does MLA save vs GQA? Concrete numbers (41.5GB → 5.1GB)?

【Multi-head Latent Attention 核心考点 3】How much memory does MLA save vs GQA? Concrete numbers (41.5GB → 5.1GB)?。技术解析：Definition: MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector c_t^{KV} \in \mathbb{R}^{d_c} (d_c = 512 in DeepSeek-V2), and per-head K/V are reconstructed by up-pro... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/mla-attention

Q: How does MLA cut activation memory in training (recomputing K/V at prefill)?

【Multi-head Latent Attention 核心考点 4】How does MLA cut activation memory in training (recomputing K/V at prefill)?。技术解析：Definition: MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector c_t^{KV} \in \mathbb{R}^{d_c} (d_c = 512 in DeepSeek-V2), and per-head K/V are reconstructed by up-pro... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/mla-attention

Q: Essential difference between MLA and MQA/GQA (sharing vs low-rank projection)?

【Multi-head Latent Attention 核心考点 5】Essential difference between MLA and MQA/GQA (sharing vs low-rank projection)?。技术解析：Definition: MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector c_t^{KV} \in \mathbb{R}^{d_c} (d_c = 512 in DeepSeek-V2), and per-head K/V are reconstructed by up-pro... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/mla-attention

MLA 低秩 KV 压缩

🎯Core Definition

MLA (Multi-head Latent Attention) is the low-rank KV compression introduced by DeepSeek-V2: the layer's K/V are first compressed into a low-dimensional latent vector

c_t^{KV} \in \mathbb{R}^{d_c}

(

d_c = 512

in DeepSeek-V2), and per-head K/V are reconstructed by up-projection at attention time; a small position-only rotary K (

k_t^R

) is kept separately for RoPE compatibility (decoupled RoPE).

💡Use Cases

standard in DeepSeek-V2/V3/R1, most beneficial for long context and large batch serving; by 2026 it is the reference design for long-context deployment at big companies, and "why MLA saves memory" is a must-know interview question.

⚡Key Problems Solved

GQA only shares KV heads, still caching

2 H d

values per token; MLA cuts per-token cache to

d_c

, with

d_c = 512 \ll 2 H d

— DeepSeek-V2 drops its KV cache from about 41.5GB (GQA variant) to about 5.1GB (MLA), a reduction of 80%+; during training, recomputing K/V from

c_t^{KV}

at prefill also saves substantial activation memory.

🎯5 High-Frequency Exam Points

How does MLA compress KV? Dimensionality of

c_t^{KV}

and the reconstruction process?

Why does MLA need decoupled RoPE? What role does

k_t^R

play?

How much memory does MLA save vs GQA? Concrete numbers (41.5GB → 5.1GB)?

How does MLA cut activation memory in training (recomputing K/V at prefill)?

Essential difference between MLA and MQA/GQA (sharing vs low-rank projection)?

📖 In-depth Guide:📄 transformer-architecture →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Multi-head Latent Attention"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardAttention Variants MHA/MQA/GQA Next Card →SSM & Hybrid Architectures

🔗 More LLMs Knowledge Cards

Agent & Tool Calling Alignment Tax & Preference Data Scaled Dot-Product Attention Benchmarks MMLU/GSM8K