← Back to AI Infrastructure Mind Map
🖥️ AI InfrastructureID: inference-engine

Inference Engines vLLM/SGLang/TensorRT-LLM

推理引擎 vLLM/SGLang/TensorRT-LLM
🎯Core Definition
An inference engine is the system-software layer that packages model weights, KV-cache scheduling, batching and kernel optimization into an efficient serving stack, maximizing GPU utilization, throughput and latency (TTFT/TBT) targets. Three mainstream engines compared: ① vLLM — pioneered PagedAttention (paged KV cache killing fragmentation) and Continuous Batching (iteration-level scheduling); OpenAI/HF-compatible API, largest ecosystem, works out of the box; with the same memory it serves far more concurrent requests than naive contiguous caching; supports tensor parallelism and prefill/decode disaggregation. ② SGLang — core is RadixAttention (radix-tree prefix KV reuse), cutting TTFT dramatically for long prompts, multi-turn chat and multi-tenant shared system prompts, plus finer-grained scheduling; strong for prefill-heavy and agentic multi-turn workloads; native constrained decoding. ③ TensorRT-LLM — NVIDIA's official stack: graph optimization + kernel fusion + automatic precision selection (FP16/BF16/FP8/INT4); the extreme option for single-machine latency/throughput, at the cost of graph compilation, C++ complexity, low flexibility and slow cold start. Common foundations: Continuous Batching (request-level dynamic batching), quantization backends (AWQ/FP8/INT4/GPTQ), tensor parallelism, PD disaggregation (compute-bound prefill and bandwidth-bound decode on separate nodes), and speculative decoding. Quant backend selection: FP8 (E4M3, natively supported on H100, minimal accuracy loss, serving default, roughly 2× FP16 throughput), AWQ (activation-aware weight quantization that protects salient channels by activation statistics, INT4 close to FP16 quality), INT4/INT8 (halve/quarter memory, enabling longer contexts). Key metrics: TTFT (time-to-first-token, dominated by prefill and scheduling), TBT/TPOT (time between tokens, decode throughput), aggregate tokens/s, P95 latency.
💡Use Cases
production LLM serving selection (vLLM general default / SGLang for long-context and multi-turn / TensorRT-LLM for peak performance); interview favorites: "compare the core mechanisms of the three engines", "why Continuous Batching is standard", "how to pick a quantization backend".
⚡Key Problems Solved
naive batching engines waste GPU cycles and memory via static batches and contiguous KV allocation; engines combine paged KV + iteration-level scheduling + prefix reuse to push GPU utilization from ~50-70% toward 90%+, and with quantization and PD disaggregation deliver several-fold throughput on the same hardware at lower cost — engine comparison is really about the lever mix of scheduling × memory management × kernel optimization.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of Inference Engines vLLM/SGLang/TensorRT-LLM.
Use Cases
In production engineering, what are the primary use cases for Inference Engines vLLM/SGLang/TensorRT-LLM and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Compare vLLM vs SGLang vs TensorRT-LLM core mechanisms: PagedAttention / RadixAttention / graph optimization + quantization; which workload suits each (TTFT-sensitive? throughput ceiling? flexibility?).
2
Why Continuous Batching is standard in every engine: vs static batching, iteration-level scheduling prevents fast requests being blocked by slow ones, 2-3× throughput; walk through per-step completion/insertion/preemption.
3
Quantization backend choice: FP8 (E4M3, H100-native, ~2× FP16 throughput) vs AWQ (activation-aware, INT4 near-FP16 quality) vs INT4/INT8 (half/quarter memory) precision/speed/memory trade-offs.
4
Why prefill/decode disaggregation boosts throughput and stability: prefill is compute-bound, decode bandwidth-bound, mixed they interfere; separated nodes each run optimal batch sizes — how TTFT and TBT benefit respectively.
5
Evaluate engines by metrics: define and measure TTFT / TBT / tokens/s throughput / P95; given per-GPU peak throughput and average output length, estimate sustainable concurrency and QPS.
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "Inference Engines vLLM/SGLang/TensorRT-LLM"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardSpeculative DecodingNext Card →Continuous Batching

🔗 More AI Infrastructure Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/inference-engine
中文速览与面试要点

推理引擎 vLLM/SGLang/TensorRT-LLM (Inference Engines vLLM/SGLang/TensorRT-LLM)

定义: 推理引擎 = 把 LLM 权重、KV 缓存调度、批处理、内核优化打包成高效推理服务的系统软件层,核心职责是最大化 GPU 利用率与吞吐、压低 TTFT/TBT。三主流对比: ① vLLM —— 首创 PagedAttention(分页 KV 缓存,消除碎片)与 Continuous Batching(迭代级调度),API 兼容 OpenAI/HF、生态最大、开箱即用,显存利用率与吞吐提升明显(对比 naive 连续缓存,同等显存可服务更多并发),支持张量并行/PD 分离;② SGLang —— 核心是 RadixAttention(基数树前缀 KV 复用,长 prompt、多轮对话、多租户共享系统提示场景 TTFT 显著下降)与更高频的调度优化,更擅长 Prefill 密集型与 Agent 多轮场景,提供结构化输出 (constrained decoding) 原生支持;③ TensorRT-LLM —— NVIDIA 官方,图优化 + 算子融合 + 精度自动选择(FP16/BF16/FP8/INT4),单卡/单机延迟与吞吐的极致选项,但图编译与 C++ 编程复杂度高、灵活性低、冷启动慢。共同底座: Continuous Batching(请求级动态批)、量化后端(AWQ/FP8/INT4/GPTQ)、张量并行、PD 分离(Prefill 算力密集节点与 Decode 带宽密集节点分开部署)、推测解码。量化后端选型: FP8(E4M3,H100 原生支持,精度损失最小,服务侧主流,吞吐比 FP16 约翻倍)、AWQ(激活感知的权重量化,按激活分布敏感度保护重要通道,INT4 下精度接近 FP16)、INT4/INT8(显存减半/减 4 倍,可支撑更长上下文)。评估指标: TTFT(首 token 延迟,受 prefill 与调度影响)、TBT/TPOT(每 token 间隔,decode 吞吐)、总吞吐 tokens/s、P95 延迟。 使用场景: 生产环境 LLM 服务选型(vLLM 通用首选 / SGLang 长上下文与多轮 / TensorRT-LLM 极致性能);面试高频“三个引擎的核心机制对比”“Continuous Batching 为什么标配”“量化后端怎么选”。 解决的问题: 朴素批处理引擎因静态批与连续 KV 分配导致 GPU 空闲与显存浪费;引擎层用分页 KV + 迭代级调度 + 前缀复用把 GPU 利用率从约 50-70% 提升到 90%+,配合量化与 PD 分离让同样硬件吞吐提升数倍、成本下降 —— 引擎对比的实质是“调度策略 × 显存管理 × 内核优化”三种杠杆的组合。

核心要点

  • vLLM vs SGLang vs TensorRT-LLM 核心机制对比: PagedAttention / RadixAttention / 图优化+量化;各自最适合什么负载(TTFT 敏感?吞吐极限?灵活性?)。
  • 为什么 Continuous Batching 是引擎标配: 对比静态批,迭代级调度下快请求不被慢请求阻塞,吞吐可提升 2-3×;解释完成/插入/抢占的每步逻辑。
  • 量化后端选型: FP8(E4M3, H100 原生,吞吐约 2×FP16)vs AWQ(激活感知,INT4 精度接近 FP16)vs INT4/INT8(显存减半/1/4)的精度/速度/显存取舍。
  • PD 分离为什么提升吞吐与稳定性: prefill 算力密集、decode 带宽密集,混跑互相干扰;分离后各自节点最优批大小,解释 TTFT 与 TBT 如何分别受益。
  • 用指标评估引擎: TTFT / TBT / 吞吐 tokens/s / P95 的定义与测量;给定单卡峰值吞吐与平均输出长度,估算可支撑的并发与 QPS。