M8-030M8: ML Systems, Engineering & ResearchInference Serving & DeploymentEasy
⚡Flashcard Drill➔Mastery:
Explain the Core Metrics, Latency Decomposition, and Goodput for Online Inference Serving
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Online inference serving is governed by tail latency percentiles (P95/P99), specialized LLM latency phases (Time To First Token vs. Time Per Output Token), concurrency throughput (QPS and token/s), availability SLAs, and cost-efficiency measured via Goodput—the maximum valid throughput delivered strictly within latency SLOs.
📌 Key Takeaways
- •Latency percentiles & decomposition: P50 (median) vs. P99 (tail experience); LLM decomposition: TTFT (prefill / compute-bound) vs. TPOT (decode / memory-bound).
- •Throughput & availability: Requests per second (QPS), generation tokens per second, error budgets, and SLA/SLO compliance rates.
- •Goodput & cost: Goodput defines productive throughput adhering to latency thresholds; unit economics measured via cost per 1,000 output tokens.
📐 Mathematical Derivations
Metrics Formalism & LLM Latency Breakdown:
(1) Latency Distribution & Percentiles:
- Average latency is deeply misleading because web traffic exhibits heavy-tailed, multi-modal distributions.
- P99 Latency: The 99th percentile represents the tail user experience; in microservice cascades, a single high-P99 dependency degrades the entire end-to-end response chain.
(2) LLM Latency Decomposition:
Total end-to-end inference latency is decomposed into two fundamentally distinct computational regimes:
- Time To First Token (TTFT):
- Measures duration from request arrival to generation of the first token.
- Dominated by the Prefill Phase: processes the entire prompt sequence of length in parallel.
- Compute-bound: high arithmetic intensity, saturating GPU Tensor Cores ( attention + GEMM).
- Time Per Output Token (TPOT):
- Measures inter-token generation latency during autoregressive decoding.
- Dominated by the Decode Phase: generates tokens sequentially one by one.
- Memory-bandwidth-bound: low arithmetic intensity; every token requires reading all model weights and KV cache tensors from GPU HBM ( compute per byte transferred).
(3) Throughput Metrics:
- QPS (Queries Per Second): Traditional stateless service capacity metric.
- Tokens / Second: Aggregated generation throughput: .
(4) Goodput vs. Raw Throughput:
- Raw Throughput: Maximized by packing huge batches (), which achieves high token throughput but inflates latency to dozens of seconds, violating user SLAs.
- Goodput Formalism:
Goodput measures the effective business throughput that genuinely satisfies contractual Service Level Objectives.
(1) Latency Distribution & Percentiles:
- Average latency is deeply misleading because web traffic exhibits heavy-tailed, multi-modal distributions.
- P99 Latency: The 99th percentile represents the tail user experience; in microservice cascades, a single high-P99 dependency degrades the entire end-to-end response chain.
(2) LLM Latency Decomposition:
Total end-to-end inference latency is decomposed into two fundamentally distinct computational regimes:
- Time To First Token (TTFT):
- Measures duration from request arrival to generation of the first token.
- Dominated by the Prefill Phase: processes the entire prompt sequence of length in parallel.
- Compute-bound: high arithmetic intensity, saturating GPU Tensor Cores ( attention + GEMM).
- Time Per Output Token (TPOT):
- Measures inter-token generation latency during autoregressive decoding.
- Dominated by the Decode Phase: generates tokens sequentially one by one.
- Memory-bandwidth-bound: low arithmetic intensity; every token requires reading all model weights and KV cache tensors from GPU HBM ( compute per byte transferred).
(3) Throughput Metrics:
- QPS (Queries Per Second): Traditional stateless service capacity metric.
- Tokens / Second: Aggregated generation throughput: .
(4) Goodput vs. Raw Throughput:
- Raw Throughput: Maximized by packing huge batches (), which achieves high token throughput but inflates latency to dozens of seconds, violating user SLAs.
- Goodput Formalism:
Goodput measures the effective business throughput that genuinely satisfies contractual Service Level Objectives.
🏭 Production Trade-offs
In-Depth Analysis & Engineering Trade-offs: ① P99 over average latency—in multi-tenant architectures, averaging conceals queue starvation and tail latency spikes experienced by valuable users. ② TTFT and TPOT require separate optimization paths—improving TTFT requires compute scaling, FlashAttention, and chunked prefill; improving TPOT requires memory bandwidth optimizations, tensor parallelism, speculative decoding, and FP8 KV caching. ③ Batch size: Throughput vs. Latency trade-off—increasing batch size amortizes model weight loading across queries (improving tokens/sec and lowering GPU cost), but linearly increases individual token generation latency (degrading TPOT); dynamic continuous batching balances this frontier. ④ Length-stratified benchmarking—evaluating LLM serving performance without segmenting by prompt and generation length produces garbage metrics; long-context inputs (32k tokens) naturally inflate TTFT by orders of magnitude compared to conversational 100-token queries. ⑤ Cost per million tokens as the ultimate unit economic—cloud serving profitability is governed by maximizing Goodput per dollar of GPU hardware spend. ⑥ Interview takeaway—break down the latency equation into TTFT and TPOT, map them to prefill (compute-bound) vs. decode (memory-bound) phases, and contrast raw throughput with SLA-constrained Goodput.
⚠️ Common Interview Pitfalls
- ✕Evaluating serving systems using average latency instead of P95/P99 percentiles, completely hiding tail latency degradation.
- ✕Conflating TTFT and TPOT into a single aggregate latency metric, making it impossible to identify whether prefill or decode is the bottleneck.
- ✕Optimizing solely for peak raw tokens/second by saturating batch sizes, resulting in catastrophic latency SLO violations and zero actual Goodput.
🎯 Interviewer Follow-up Inquiries
- ?How does Chunked Prefill prevent massive prefill requests from starving active autoregressive decode iterations in continuous batching engines?
- ?Why is the autoregressive decode phase strictly memory-bandwidth bound rather than compute bound on modern GPUs?
- ?How is dynamic request batching tuned to maximize Goodput without exceeding P99 latency SLO guardrails?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚