Inference Engines vLLM/SGLang/TensorRT-LLM

An inference engine is the system-software layer that packages model weights, KV-cache scheduling, batching and kernel optimization into an efficient serving stack, maximizing GPU utilization, throughput and latency (TTFT/TBT) targets. Three mainstream engines compared: ① vLLM — pioneered PagedAttention (paged KV cache killing fragmentation) and Continuous Batching (iteration-level scheduling); OpenAI/HF-compatible API, largest ecosystem, works out of the box; with the same memory it serves far more concurrent requests than naive contiguous caching; supports tensor parallelism and prefill/decode disaggregation. ② SGLang — core is RadixAttention (radix-tree prefix KV reuse), cutting TTFT dramatically for long prompts, multi-turn chat and multi-tenant shared system prompts, plus finer-grained scheduling; strong for prefill-heavy and agentic multi-turn workloads; native constrained decoding. ③ TensorRT-LLM — NVIDIA's official stack: graph optimization + kernel fusion + automatic precision selection (FP16/BF16/FP8/INT4); the extreme option for single-machine latency/throughput, at the cost of graph compilation, C++ complexity, low flexibility and slow cold start. Common foundations: Continuous Batching (request-level dynamic batching), quantization backends (AWQ/FP8/INT4/GPTQ), tensor parallelism, PD disaggregation (compute-bound prefill and bandwidth-bound decode on separate nodes), and speculative decoding. Quant backend selection: FP8 (E4M3, natively supported on H100, minimal accuracy loss, serving default, roughly 2× FP16 throughput), AWQ (activation-aware weight quantization that protects salient channels by activation statistics, INT4 close to FP16 quality), INT4/INT8 (halve/quarter memory, enabling longer contexts). Key metrics: TTFT (time-to-first-token, dominated by prefill and scheduling), TBT/TPOT (time between tokens, decode throughput), aggregate tokens/s, P95 latency.

production LLM serving selection (vLLM general default / SGLang for long-context and multi-turn / TensorRT-LLM for peak performance); interview favorites: "compare the core mechanisms of the three engines", "why Continuous Batching is standard", "how to pick a quantization backend".

naive batching engines waste GPU cycles and memory via static batches and contiguous KV allocation; engines combine paged KV + iteration-level scheduling + prefix reuse to push GPU utilization from ~50-70% toward 90%+, and with quantization and PD disaggregation deliver several-fold throughput on the same hardware at lower cost — engine comparison is really about the lever mix of scheduling × memory management × kernel optimization.

Test Your Knowledge: Practice Questions for "Inference Engines vLLM/SGLang/TensorRT-LLM"

Single choice pitfall questions with instant feedback and mistake tracking.