⚡ LLMsID: kv-cache-management

KV Cache Management

Q: Meaning of each term in $2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b$? A concrete worked example?

【KV Cache Management 核心考点 1】Meaning of each term in $2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b$? A concrete worked example?。技术解析：Definition: at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size = 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/kv-cache-management

Q: Why does the KV cache exist only at inference, not training (parallel full-sequence training vs sequential generation)?

【KV Cache Management 核心考点 2】Why does the KV cache exist only at inference, not training (parallel full-sequence training vs sequential generation)?。技术解析：Definition: at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size = 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/kv-cache-management

Q: How PagedAttention works? How do block size and page tables reduce fragmentation?

【KV Cache Management 核心考点 3】How PagedAttention works? How do block size and page tables reduce fragmentation?。技术解析：Definition: at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size = 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/kv-cache-management

Q: What are StreamingLLM's attention sinks, and why are the first tokens special?

【KV Cache Management 核心考点 4】What are StreamingLLM's attention sinks, and why are the first tokens special?。技术解析：Definition: at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size = 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/kv-cache-management

Q: How much do GQA, MLA and INT4 quantization each shrink the KV cache (687 → 86GB etc.)?

【KV Cache Management 核心考点 5】How much do GQA, MLA and INT4 quantization each shrink the KV cache (687 → 86GB etc.)?。技术解析：Definition: at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size = 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b (2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); ... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/kv-cache-management

KV Cache 管理

🎯Core Definition

at autoregressive decode time every new token must attend to all history, so past K/V are cached to avoid recomputation: cache size

= 2 \cdot 2 \cdot L \cdot H \cdot d \cdot s \cdot b

(2 bytes per fp16 element × K and V × layers × heads × head dim × seq len × batch); LLaMA-3 70B at

s = 8192

b = 32

needs about 687GB — far exceeding its 140GB of weights — and about 86GB with GQA (8 groups). Management techniques: PagedAttention (vLLM) manages the cache via page tables in 16-token blocks to remove fragmentation; StreamingLLM uses attention sinks plus a rolling window; PD separation runs prefill and decode on separate instances.

💡Use Cases

every autoregressive inference and long-context service; a favorite interview exercise is estimating KV memory from layers/heads/dim/seq and comparing the impact of GQA/MLA/quantization (cross-module — see the AI_Infra KV Cache & PagedAttention guide).

⚡Key Problems Solved

the cache grows linearly with sequence length and is the #1 serving memory cost, while fixed pre-allocation wastes 50-60%+ to fragmentation; PagedAttention decouples occupancy from position, cutting fragmentation to near zero and boosting throughput 2-4x (up to 24x vs FasterTransformer in the vLLM paper); StreamingLLM bounds per-layer cache to a fixed window for very long streaming dialogues; PD separation keeps compute-heavy prefill from stalling decode.

🎯5 High-Frequency Exam Points

Meaning of each term in