Inference Engines vLLM/SGLang/TensorRT-LLM

An inference engine is the system-software layer that packages model weights, KV-cache scheduling, batching and kernel optimization into an efficient serving stack, maximizing GPU utilization, throughput and latency (TTFT/TBT) targets. Three mainstream engines compared: ① vLLM — pioneered PagedAttention (paged KV cache killing fragmentation) and Continuous Batching (iteration-level scheduling); OpenAI/HF-compatible API, largest ecosystem, works out of the box; with the same memory it serves far more concurrent requests than naive contiguous caching; supports tensor parallelism and prefill/decode disaggregation. ② SGLang — core is RadixAttention (radix-tree prefix KV reuse), cutting TTFT dramatically for long prompts, multi-turn chat and multi-tenant shared system prompts, plus finer-grained scheduling; strong for prefill-heavy and agentic multi-turn workloads; native constrained decoding. ③ TensorRT-LLM — NVIDIA's official stack: graph optimization + kernel fusion + automatic precision selection (FP16/BF16/FP8/INT4); the extreme option for single-machine latency/throughput, at the cost of graph compilation, C++ complexity, low flexibility and slow cold start. Common foundations: Continuous Batching (request-level dynamic batching), quantization backends (AWQ/FP8/INT4/GPTQ), tensor parallelism, PD disaggregation (compute-bound prefill and bandwidth-bound decode on separate nodes), and speculative decoding. Quant backend selection: FP8 (E4M3, natively supported on H100, minimal accuracy loss, serving default, roughly 2× FP16 throughput), AWQ (activation-aware weight quantization that protects salient channels by activation statistics, INT4 close to FP16 quality), INT4/INT8 (halve/quarter memory, enabling longer contexts). Key metrics: TTFT (time-to-first-token, dominated by prefill and scheduling), TBT/TPOT (time between tokens, decode throughput), aggregate tokens/s, P95 latency.

production LLM serving selection (vLLM general default / SGLang for long-context and multi-turn / TensorRT-LLM for peak performance); interview favorites: "compare the core mechanisms of the three engines", "why Continuous Batching is standard", "how to pick a quantization backend".

naive batching engines waste GPU cycles and memory via static batches and contiguous KV allocation; engines combine paged KV + iteration-level scheduling + prefix reuse to push GPU utilization from ~50-70% toward 90%+, and with quantization and PD disaggregation deliver several-fold throughput on the same hardware at lower cost — engine comparison is really about the lever mix of scheduling × memory management × kernel optimization.

Compare vLLM vs SGLang vs TensorRT-LLM core mechanisms: PagedAttention / RadixAttention / graph optimization + quantization; which workload suits each (TTFT-sensitive? throughput ceiling? flexibility?).

Why Continuous Batching is standard in every engine: vs static batching, iteration-level scheduling prevents fast requests being blocked by slow ones, 2-3× throughput; walk through per-step completion/insertion/preemption.

Quantization backend choice: FP8 (E4M3, H100-native, ~2× FP16 throughput) vs AWQ (activation-aware, INT4 near-FP16 quality) vs INT4/INT8 (half/quarter memory) precision/speed/memory trade-offs.

Why prefill/decode disaggregation boosts throughput and stability: prefill is compute-bound, decode bandwidth-bound, mixed they interfere; separated nodes each run optimal batch sizes — how TTFT and TBT benefit respectively.

Evaluate engines by metrics: define and measure TTFT / TBT / tokens/s throughput / P95; given per-GPU peak throughput and average output length, estimate sustainable concurrency and QPS.