Continuous Batching

Continuous Batching (iteration-level scheduling, introduced by Orca) does not wait for an entire batch to finish before serving the next one; at the end of every decoding iteration (one token step) the batch membership is reshuffled: requests that finished (naturally or by max-length truncation) are evicted, waiting requests that completed prefill are inserted, and each in-flight request keeps its own KV-cache progress. Contrast static (request-level) batching, which pins a batch until all requests complete — the slowest request (e.g. 40 tokens) blocks the fastest, whose slot idles the rest of the time → low GPU utilization. With continuous batching every step tries to keep the batch full, raising throughput roughly 2-3× (Orca reports about 2.5×; the win is biggest for mixed long/short workloads, since short requests finish fast, free slots, and new ones fill in). Key implementation points: ① a new request must prefill first (compute the whole prompt's KV in one pass) before joining decode iterations; how prefill and decode are co-scheduled decides TTFT; ② preemption: under memory pressure the engine can preempt some requests' decode slots (common policies: new-prefill admission or priority-based eviction); evicted requests' KV is swapped to CPU or discarded to be restarted/recomputed later; ③ sequences of unequal length in the batch need padding or chunked attention to avoid wasted compute; ④ it composes naturally with PagedAttention, whose paged KV allows slots to be allocated/freed block-by-block as requests come and go, fragmentation-free.

the default scheduler in every mainstream engine (vLLM/SGLang/TensorRT-LLM); biggest gains for online chat/API traffic (random arrival, mixed lengths); interview staples: "static vs continuous batching", "preemption and swap", "why GPU utilization goes up".

static batching wastes GPU cycles from two sources — the slowest request in a batch blocks everyone (bubble), and waits between batch arrivals. Continuous batching refines the scheduling granularity from request to iteration, so every step has finished requests freeing slots and new requests filling them, driving compute density toward 100%; on the same hardware this is roughly 2-3× more throughput, with more controllable TTFT since new requests no longer wait for a whole batch to drain.

Static vs continuous batching: draw a timeline with two requests (10 tokens vs 40 tokens) showing static batching's idle time vs continuous batching's refill, and why throughput rises 2-3×.

Iteration-level scheduling: what the engine does at the end of each decode iteration (evict finished, insert prefilled newcomers, dynamic membership); why a request must prefill before joining decode.

Preemption and swap: who gets preempted under memory pressure (new prefill vs running decode), where evicted KV goes (CPU swap / discard-restart), and each option's latency cost.

Why workloads with many short requests benefit most: fast completion, fast slot turnover; relate to the padding/chunked-attention cost of unequal-length sequences in a batch.

How continuous batching composes with PagedAttention and PD disaggregation: paged KV makes dynamic membership fragmentation-free; how scheduling works in prefill nodes vs decode nodes.