Prefill/Decode Disaggregation

Prefill/Decode disaggregation (PD split) = running the two phases of a request on different instance types: Prefill processes the whole prompt in parallel — matrix-matrix (GEMM) work that is compute-bound with high GPU utilization; Decode generates one token at a time — a matrix-vector (GEMV) step per token that is bandwidth-bound, reading the full weight set to emit a single token, with low utilization. After the split, Prefill instances turn prompts into KV caches and ship them over high-speed networks (RDMA/InfiniBand) to Decode instances, which focus purely on low-latency batched generation. Scheduling: the two pools scale independently; Decode instances are load-balanced by occupied KV size / sequence length (not connection count), with token-level scheduling and pipelined KV transfer.

high-concurrency online serving (chat/agent workloads mixing long prompts and long outputs) that must keep both TTFT and ITL low; vLLM/SGLang have native PD support and top vendors like DeepSeek run this architecture in production; interview favorites: why Prefill and Decode have different bottlenecks, and what PD split solves.

when colocated, Prefill bursts (large batches of big prompts) saturate compute and wreck the TTFT/ITL of in-flight Decode requests — the two loads interfere and their scheduling goals conflict. Splitting them: ① each pool is provisioned and scaled to its own bottleneck (compute vs bandwidth), raising GPU utilization; ② TTFT is decided by Prefill instances, ITL by Decode instances, so latency targets are independently tunable; ③ bursts are absorbed by the Prefill pool while Decode stays stable — at the cost of KV transfer over the network and more complex scheduling.

Why is Prefill compute-bound while Decode is bandwidth-bound (GEMM vs GEMV — reading all weights per step to emit one token)? Explain with a Roofline model.

PD split architecture: how does the KV cache travel from Prefill to Decode instances (network / async pipelining); what new bottleneck does the split introduce?

Why is Decode-instance load balancing based on occupied KV size / sequence length instead of connection or request counts? How do you avoid imbalance with mixed long and short requests?

Which phase decides TTFT vs ITL (TPOT)? How does PD split let you optimize them independently, and why do they fight each other when colocated?

Relation between PD split, continuous batching and chunked prefill: why is chunked prefill a "soft disaggregation" alternative, and what are the trade-offs of each?