M8-030M8: ML Systems, Engineering & ResearchInference Serving & DeploymentEasy
⚡Flashcard Drill➔Mastery:

Explain the Core Metrics, Latency Decomposition, and Goodput for Online Inference Serving

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M8-031 →
📐 Mathematical Definition
latency: TTFT,TPOT;throughput: QPS;goodput=f(SLO)\text{latency}:\ \text{TTFT},\text{TPOT};\qquad \text{throughput}:\ \text{QPS};\qquad \text{goodput}=f(\text{SLO})
⚡ One-Sentence Core Answer
Online inference serving is governed by tail latency percentiles (P95/P99), specialized LLM latency phases (Time To First Token vs. Time Per Output Token), concurrency throughput (QPS and token/s), availability SLAs, and cost-efficiency measured via Goodput—the maximum valid throughput delivered strictly within latency SLOs.

📌 Key Takeaways

  • •
    Latency percentiles & decomposition: P50 (median) vs. P99 (tail experience); LLM decomposition: TTFT (prefill / compute-bound) vs. TPOT (decode / memory-bound).
  • •
    Throughput & availability: Requests per second (QPS), generation tokens per second, error budgets, and SLA/SLO compliance rates.
  • •
    Goodput & cost: Goodput defines productive throughput adhering to latency thresholds; unit economics measured via cost per 1,000 output tokens.

📐 Mathematical Derivations

Metrics Formalism & LLM Latency Breakdown:

(1) Latency Distribution & Percentiles:
- Average latency is deeply misleading because web traffic exhibits heavy-tailed, multi-modal distributions.
- P99 Latency: The 99th percentile represents the tail user experience; in microservice cascades, a single high-P99 dependency degrades the entire end-to-end response chain.

(2) LLM Latency Decomposition:
Total end-to-end inference latency is decomposed into two fundamentally distinct computational regimes:
Ttotal=Tqueue+TTFT+TPOT×(Ntokens−1)T_{\text{total}} = T_{\text{queue}} + \text{TTFT} + \text{TPOT} \times (N_{\text{tokens}} - 1)
- Time To First Token (TTFT):
- Measures duration from request arrival to generation of the first token.
- Dominated by the Prefill Phase: processes the entire prompt sequence of length SinS_{\text{in}} in parallel.
- Compute-bound: high arithmetic intensity, saturating GPU Tensor Cores (O(Sin2)O(S_{\text{in}}^2) attention + GEMM).
- Time Per Output Token (TPOT):
- Measures inter-token generation latency during autoregressive decoding.
- Dominated by the Decode Phase: generates tokens sequentially one by one.
- Memory-bandwidth-bound: low arithmetic intensity; every token requires reading all model weights and KV cache tensors from GPU HBM (O(1)O(1) compute per byte transferred).

(3) Throughput Metrics:
- QPS (Queries Per Second): Traditional stateless service capacity metric.
- Tokens / Second: Aggregated generation throughput: TPS=∑i=1BTokensiΔt\text{TPS} = \sum_{i=1}^B \frac{\text{Tokens}_i}{\Delta t}.

(4) Goodput vs. Raw Throughput:
- Raw Throughput: Maximized by packing huge batches (B=256B=256), which achieves high token throughput but inflates latency to dozens of seconds, violating user SLAs.
- Goodput Formalism:
Goodput=∑i=1MI(Latencyi≤SLOlat∧Statusi=200)×Throughputi\text{Goodput} = \sum_{i=1}^M \mathbb{I}(\text{Latency}_i \le \text{SLO}_{\text{lat}} \land \text{Status}_i = 200) \times \text{Throughput}_i
Goodput measures the effective business throughput that genuinely satisfies contractual Service Level Objectives.

🏭 Production Trade-offs

In-Depth Analysis & Engineering Trade-offs: ① P99 over average latency—in multi-tenant architectures, averaging conceals queue starvation and tail latency spikes experienced by valuable users. ② TTFT and TPOT require separate optimization paths—improving TTFT requires compute scaling, FlashAttention, and chunked prefill; improving TPOT requires memory bandwidth optimizations, tensor parallelism, speculative decoding, and FP8 KV caching. ③ Batch size: Throughput vs. Latency trade-off—increasing batch size amortizes model weight loading across queries (improving tokens/sec and lowering GPU cost), but linearly increases individual token generation latency (degrading TPOT); dynamic continuous batching balances this frontier. ④ Length-stratified benchmarking—evaluating LLM serving performance without segmenting by prompt and generation length produces garbage metrics; long-context inputs (32k tokens) naturally inflate TTFT by orders of magnitude compared to conversational 100-token queries. ⑤ Cost per million tokens as the ultimate unit economic—cloud serving profitability is governed by maximizing Goodput per dollar of GPU hardware spend. ⑥ Interview takeaway—break down the latency equation into TTFT and TPOT, map them to prefill (compute-bound) vs. decode (memory-bound) phases, and contrast raw throughput with SLA-constrained Goodput.
⚠️ Common Interview Pitfalls
  • ✕
    Evaluating serving systems using average latency instead of P95/P99 percentiles, completely hiding tail latency degradation.
  • ✕
    Conflating TTFT and TPOT into a single aggregate latency metric, making it impossible to identify whether prefill or decode is the bottleneck.
  • ✕
    Optimizing solely for peak raw tokens/second by saturating batch sizes, resulting in catastrophic latency SLO violations and zero actual Goodput.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Chunked Prefill prevent massive prefill requests from starving active autoregressive decode iterations in continuous batching engines?
  • ?
    Why is the autoregressive decode phase strictly memory-bandwidth bound rather than compute bound on modern GPUs?
  • ?
    How is dynamic request batching tuned to maximize Goodput without exceeding P99 latency SLO guardrails?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM8-029: Explain Engineering Strategies and Levers for Machine Learning Training Cost Optimization📋 Back to BankNext →M8-031: Explain Common Model Serving Deployment Paradigms and Architectural Patterns