M4-058M4: Sequences & TransformersKV Cache & Inference OptimizationsEasy
⚡Flashcard Drill➔Mastery:

Differences Between Prefill and Decode Phases in LLM Inference

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-059 →
📐 Mathematical Definition
prefill:L queries parallel, compute-bound;decode:1 query, memory-bound\text{prefill}: L\ \text{queries parallel},\ \text{compute-bound};\qquad \text{decode}: 1\ \text{query},\ \text{memory-bound}
⚡ One-Sentence Core Answer
The prefill phase processes the prompt in parallel and is compute-bound, whereas the decode phase generates tokens autoregressively one by one and is memory-bandwidth bound.

📌 Key Takeaways

  • •
    Prefill (Time-to-First-Token / TTFT): computes Q,K,VQ, K, V for all prompt tokens simultaneously in parallel; high arithmetic intensity, saturating GPU Tensor Cores (compute-bound)
  • •
    Decode (Time-Per-Output-Token / TPOT): generates one token per step (QQ length is 1); reads the entire historical KV cache for a tiny matrix-vector multiplication, severely constrained by memory bandwidth (memory-bound)
  • •
    Different optimization objectives: Prefill prioritizes FlashAttention, tensor parallelism, and prompt caching; Decode prioritizes batching concurrency, GQA, quantization, and speculative decoding

📐 Mathematical Derivations

Mathematical Mechanism: Prefill Phase: Given prompt length LinL_{\text{in}}, the input is X∈RLin×dX \in \mathbb{R}^{L_{\text{in}} \times d}. Q,K,V∈RLin×dQ, K, V \in \mathbb{R}^{L_{\text{in}} \times d} are computed in a single forward pass. The attention computation involves GEMM (General Matrix Multiply): QKT∈RLin×LinQ K^T \in \mathbb{R}^{L_{\text{in}} \times L_{\text{in}}}. The arithmetic intensity is approximately 2Lind+2Lin2d2d2+2Lind\frac{2 L_{\text{in}} d + 2 L_{\text{in}}^2 d}{2 d^2 + 2 L_{\text{in}} d}, which scales with LinL_{\text{in}}. For moderate to large LinL_{\text{in}} (e.g., >512>512), operational intensity surpasses the GPU hardware balance point (e.g., ~150 FLOPs/byte on A100), placing prefill squarely in the compute-bound regime. Decode Phase: At generation step tt, the query is a single vector qt∈R1×dq_t \in \mathbb{R}^{1 \times d}. The model must load the model weights (W∈Rd×dW \in \mathbb{R}^{d \times d}) and all cached keys and values K1:t,V1:t∈Rt×dK_{1:t}, V_{1:t} \in \mathbb{R}^{t \times d} from HBM. The operations are GEMV (General Matrix-Vector Multiply). FLOPs executed are O(d2+td)O(d^2 + t d), while bytes transferred are O(d2+td)O(d^2 + t d). The arithmetic intensity is ≈1\approx 1 FLOP/byte, orders of magnitude below hardware compute capability (~312 TFLOPs vs 2 TB/s HBM bandwidth), making decode strictly memory bandwidth-bound.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Interference in Co-located Serving: When prefill requests (bursty, high compute, monopolizing GPU cores) and decode requests (latency-sensitive, low compute, high bandwidth) share the same GPU, prefill execution causes massive latency spikes (TPOT jitter) for ongoing decode requests. ② Mitigation: Chunked Prefill: Splitting long prompt prefills into smaller chunks (e.g., 512 tokens) and co-scheduling them alongside decode batches to smooth out GPU execution. ③ Architectural Evolution: PD Disaggregation: Physically separating prefill instances and decode instances onto dedicated GPU clusters, transferring KV caches over high-speed networks (InfiniBand/PCIe Gen5) to achieve optimal hardware utilization. ④ Batching Impact: Increasing batch size during decode amortizes weight loading across multiple requests, converting GEMV toward GEMM and boosting throughput at the cost of higher latency. ⑤ Interview Strategy: Frame the answer around the Roofline Model, contrasting TTFT vs. TPOT metrics and highlighting system-level scheduling solutions like vLLM and TensorRT-LLM.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming both prefill and decode phases are compute-bound or that adding more compute always speeds up decoding
  • ✕
    Failing to distinguish TTFT from TPOT when discussing latency SLAs
  • ✕
    Ignoring the severe GPU resource contention caused by running prefill and decode concurrently on the same device
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Chunked Prefill improve decode latency jitter?
  • ?
    Under what conditions does PD Disaggregation justify the network overhead of KV cache transfer?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-057: KV Cache Principles and Why It Cannot Be Used During Training📋 Back to BankNext →M4-059: Losslessness and Theoretical Guarantee of Speculative Decoding