← Back to AI Infrastructure Mind Map
🖥️ AI InfrastructureID: kv-cache-paged

KV Cache & PagedAttention

KV Cache 与 PagedAttention
🎯Core Definition
KV Cache is the inference memory structure that caches the Key/Value vectors of already-generated tokens during autoregressive decoding, avoiding recomputing attention over the full history at every step. Each decoding step only computes the new token's Q, attends against cached K/V, and appends new K/V to the cache. Memory formula (FP16, 2 bytes/element): KV Bytes=2⋅2⋅nlayers⋅nkv_heads⋅dhead⋅s⋅b\text{KV Bytes} = 2 \cdot 2 \cdot n_{layers} \cdot n_{kv\_heads} \cdot d_{head} \cdot s \cdot b — the first 2 = one K and one V, the second 2 = 2 bytes per FP16 element, ss = sequence length, bb = batch size. Worked example (LLaMA-3 70B: 80 layers, 8 GQA KV heads, dhead=128d_{head}=128, 32K context, batch 8): 2×2×80×8×128×32768×8≈85.9 GB≈86 GB2 \times 2 \times 80 \times 8 \times 128 \times 32768 \times 8 \approx 85.9\text{ GB} \approx 86\text{ GB}; the same model with MHA (64 KV heads) needs about 687 GB687\text{ GB} — exactly 64/8=8×64/8 = 8\times more, the direct payoff of GQA. PagedAttention (vLLM): instead of allocating KV as one contiguous array sized for worst case, KV is sliced into fixed-size blocks (default 16 tokens per block, i.e. 16×nkv_heads×dhead16 \times n_{kv\_heads} \times d_{head} elements) and a block table maps logical positions to physical blocks, paging like an OS. This eliminates internal fragmentation (static pre-allocation exceeding actual generated length) and external fragmentation (interleaved sequences of different lengths leaving unusable gaps), pushing effective memory utilization from roughly 60-80% for contiguous allocation toward ~100%.
💡Use Cases
standard in every LLM serving engine (vLLM/SGLang/TensorRT-LLM internals); the dominant memory bottleneck for long-context and high-concurrency batched decode; interview staples: "hand-compute KV cache memory", "why PagedAttention saves memory", "how GQA/MLA compress it".
⚡Key Problems Solved
without caching, each step's attention is O(s)O(s) and total decode compute is O(s2)O(s^2) with massive recomputation; KV caching makes each step O(1)O(1) incremental, but memory grows linearly with s×bs \times b (tens of GB at long context / high concurrency, 30-60% of a GPU's memory). PagedAttention removes fragmentation and approaches 100% utilization, letting a 70B model serve 16K-context requests in batches; GQA/MLA shrink nkv_headsn_{kv\_heads} by 8× or more, cutting KV memory linearly.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of KV Cache & PagedAttention.
Use Cases
In production engineering, what are the primary use cases for KV Cache & PagedAttention and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Hand-compute KV cache memory: write KV Bytes=2⋅2⋅nlayers⋅nkv_heads⋅dhead⋅s⋅b\text{KV Bytes} = 2\cdot2\cdot n_{layers}\cdot n_{kv\_heads}\cdot d_{head}\cdot s\cdot b and explain what each 2 means; for 80 layers, 8 KV heads, dhead=128d_{head}=128, 16K context, batch 16, compute per-sequence and total GB.
2
Why PagedAttention removes memory fragmentation: 16-token blocks plus a block table; how internal and external fragmentation arise and disappear; utilization improvement from ~60-80% to ~100%.
3
How GQA/MLA compress KV memory: which term in the formula shrinks when MHA → GQA shares KV heads; LLaMA-3 70B (64→8) gives an 8× reduction; MLA's low-rank projection idea.
4
KV write patterns in prefill vs decode: prefill writes all s KV entries at once, decode appends one per step; why the memory peak sits at the end of prefill.
5
Precision trade-offs of KV quantization: why quantizing KV (e.g. INT8) is more sensitive than weight quantization; how it composes with FlashAttention and streaming/sliding-window KV to cap memory.
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "KV Cache & PagedAttention"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardCheckpointing & RecoveryNext Card →Speculative Decoding

🔗 More AI Infrastructure Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/kv-cache-paged
中文速览与面试要点

KV Cache 与 PagedAttention (KV Cache & PagedAttention)

定义: KV Cache = 自回归解码时把已生成 token 的 Key/Value 向量缓存起来、避免每步重复计算历史 attention 的推理显存结构。每步解码只需计算新 token 的 Q,与缓存的历史 K/V 做 attention,新 K/V 追加进缓存。缓存显存公式(每元素 FP16 占 2 字节): KV Bytes = 2 2 n_layers n_kv\_heads d_head s b —— 第 1 个 2 = K 与 V 各一份,第 2 个 2 = FP16 每元素 2 字节,s = 序列长度,b = batch。数值例 (LLaMA-3 70B: 80 层、GQA 8 个 KV heads、d_head=128,32K 上下文、8 并发): 2 2 80 8 128 32768 8 85.9 GB 86 GB;若换回同规模 MHA(64 个 KV heads)则约 687 GB —— 正好差 64/8 = 8 倍,这就是 GQA 压缩 KV 的直观收益。PagedAttention (vLLM): KV 不再按连续长数组预分配,而是切成固定大小的 block(默认每块 16 个 token,即 16 n_kv\_heads d_head 个元素),用页表 (block table) 记录逻辑位置 → 物理块映射,像操作系统分页一样按需分配: 消除内部碎片(静态预分配超出实际生成长度)与外部碎片(不同长度序列交错、先来后走导致空闲空间无法拼接),并把「预留显存」降到几乎为 0,显存利用率从连续分配的约 60-80% 提升到接近 100%。 使用场景: 所有 LLM 推理服务标配(vLLM/SGLang/TensorRT-LLM 内核),长上下文、高并发批量解码是显存主瓶颈;面试高频“手算 KV cache 显存”“PagedAttention 为什么省显存”“GQA/MLA 如何压缩”。 解决的问题: attention 每步与全部历史计算复杂度为 O(s),不缓存则总计算量 O(s^2) 且重复;KV 缓存把解码复杂度降到每步 O(1) 增量计算,但显存随 s b 线性增长(长序列/大并发下可达数十 GB,占单卡显存 30-60%)。PagedAttention 用分页消除碎片、逼近 100% 显存利用率,使 70B 模型 16K 上下文能批量服务;GQA/MLA 进一步把 n_kv\_heads 压缩 8 倍/更低,直接线性降低 KV 显存。

核心要点

  • 手算 KV cache 显存: 写出公式 KV Bytes = 2 2 n_layers n_kv\_heads d_head s b 并解释两个 2 的含义;给定 80 层、8 个 KV heads、d_head=128、16K 上下文、batch 16,算每序列/整批多少 GB。
  • PagedAttention 为什么能消除显存碎片: 16 token/块的页表机制;内部碎片与外部碎片分别怎么产生、分页后如何消失;显存利用率从约多少提升到多少。
  • GQA/MLA 如何压缩 KV 显存: MHA → GQA 共享 KV heads 后公式里哪一项变小;LLaMA-3 70B (64→8) 对应 8 倍降幅;MLA 更进一步(低秩投影压缩)的原理。
  • Prefill 与 Decode 阶段 KV 缓存的写入与复用差异: prefill 一次性写入全部 s 个 KV、decode 每步追加一个;为什么长 prompt 的显存峰值出现在 prefill 末尾。
  • KV 缓存与量化/压缩的精度权衡: 为什么 KV 量化 (如 INT8) 比权重量化更敏感;与 FlashAttention、StreamingLLM 滑动窗口如何组合控制显存。