M4-058M4: Sequences & TransformersKV Cache & Inference OptimizationsEasy
⚡Flashcard Drill➔Mastery:
Differences Between Prefill and Decode Phases in LLM Inference
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The prefill phase processes the prompt in parallel and is compute-bound, whereas the decode phase generates tokens autoregressively one by one and is memory-bandwidth bound.
📌 Key Takeaways
- •Prefill (Time-to-First-Token / TTFT): computes for all prompt tokens simultaneously in parallel; high arithmetic intensity, saturating GPU Tensor Cores (compute-bound)
- •Decode (Time-Per-Output-Token / TPOT): generates one token per step ( length is 1); reads the entire historical KV cache for a tiny matrix-vector multiplication, severely constrained by memory bandwidth (memory-bound)
- •Different optimization objectives: Prefill prioritizes FlashAttention, tensor parallelism, and prompt caching; Decode prioritizes batching concurrency, GQA, quantization, and speculative decoding
📐 Mathematical Derivations
Mathematical Mechanism: Prefill Phase: Given prompt length , the input is . are computed in a single forward pass. The attention computation involves GEMM (General Matrix Multiply): . The arithmetic intensity is approximately , which scales with . For moderate to large (e.g., ), operational intensity surpasses the GPU hardware balance point (e.g., ~150 FLOPs/byte on A100), placing prefill squarely in the compute-bound regime. Decode Phase: At generation step , the query is a single vector . The model must load the model weights () and all cached keys and values from HBM. The operations are GEMV (General Matrix-Vector Multiply). FLOPs executed are , while bytes transferred are . The arithmetic intensity is FLOP/byte, orders of magnitude below hardware compute capability (~312 TFLOPs vs 2 TB/s HBM bandwidth), making decode strictly memory bandwidth-bound.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Interference in Co-located Serving: When prefill requests (bursty, high compute, monopolizing GPU cores) and decode requests (latency-sensitive, low compute, high bandwidth) share the same GPU, prefill execution causes massive latency spikes (TPOT jitter) for ongoing decode requests. ② Mitigation: Chunked Prefill: Splitting long prompt prefills into smaller chunks (e.g., 512 tokens) and co-scheduling them alongside decode batches to smooth out GPU execution. ③ Architectural Evolution: PD Disaggregation: Physically separating prefill instances and decode instances onto dedicated GPU clusters, transferring KV caches over high-speed networks (InfiniBand/PCIe Gen5) to achieve optimal hardware utilization. ④ Batching Impact: Increasing batch size during decode amortizes weight loading across multiple requests, converting GEMV toward GEMM and boosting throughput at the cost of higher latency. ⑤ Interview Strategy: Frame the answer around the Roofline Model, contrasting TTFT vs. TPOT metrics and highlighting system-level scheduling solutions like vLLM and TensorRT-LLM.
⚠️ Common Interview Pitfalls
- ✕Assuming both prefill and decode phases are compute-bound or that adding more compute always speeds up decoding
- ✕Failing to distinguish TTFT from TPOT when discussing latency SLAs
- ✕Ignoring the severe GPU resource contention caused by running prefill and decode concurrently on the same device
🎯 Interviewer Follow-up Inquiries
- ?How does Chunked Prefill improve decode latency jitter?
- ?Under what conditions does PD Disaggregation justify the network overhead of KV cache transfer?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚