M4-111M4: Sequences & TransformersQuantization & AccelerationHard
⚡Flashcard Drill➔Mastery:

Systematic Metric Framework for Evaluating Inference Optimizations: TTFT, TPOT, MFU, and Cost

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-112 →
📐 Mathematical Definition
eval: bottleneck→metrics (TTFT/TPOT/throughput/goodput)→quality→ablation\text{eval}:\ \text{bottleneck}\to\text{metrics (TTFT/TPOT/throughput/goodput)}\to\text{quality}\to\text{ablation}
⚡ One-Sentence Core Answer
A rigorous inference optimization evaluation framework diagnoses hardware bottlenecks via profiling, verifies end-to-end latency SLAs and Goodput, validates downstream task accuracy, and quantifies unit cost economics.

📌 Key Takeaways

  • •
    Four-step evaluation framework: 1) Hardware diagnosis (Roofline & profiling); 2) Serving metrics (TTFT, TPOT, Throughput, Goodput within SLO); 3) Quality verification (accuracy & perplexity); 4) Stability & P99 tail latency
  • •
    Goodput vs Throughput: raw throughput counts all processed tokens; Goodput counts only tokens delivered strictly within latency Service Level Objectives (SLOs)
  • •
    Amdahl's Law in serving: optimizations must target the proven hardware bottleneck (memory bandwidth vs compute); optimizing non-bottleneck components yields negligible returns

📐 Mathematical Derivations

Mathematical Mechanism: 1. Latency & Goodput Metrics: - TTFT (Time-to-First-Token): Evaluates prefill responsiveness: TTFT=Tfirst_token−Tarrival=Tqueue+Tprefill\text{TTFT} = T_{\text{first\_token}} - T_{\text{arrival}} = T_{\text{queue}} + T_{\text{prefill}} - TPOT (Time-Per-Output-Token): Evaluates decoding fluency: TPOT=Ttotal−TTFTNgenerated_tokens−1\text{TPOT} = \frac{T_{\text{total}} - \text{TTFT}}{N_{\text{generated\_tokens}} - 1} - Goodput (SLO-Compliant Throughput): Goodput=1Twindow∑i∈RsuccessNi⋅I(TTFTi≤SLOTTFT∧P99_TPOTi≤SLOTPOT)\text{Goodput} = \frac{1}{T_{\text{window}}} \sum_{i \in \mathcal{R}_{\text{success}}} N_i \cdot \mathbb{I}(\text{TTFT}_i \le \text{SLO}_{\text{TTFT}} \land \text{P99\_TPOT}_i \le \text{SLO}_{\text{TPOT}}) 2. Hardware Efficiency Metrics: - MFU (Model FLOPs Utilization): MFU=Theoretical FLOPs per Token×Throughput [tokens/s]Peak Hardware FLOPs Capability\text{MFU} = \frac{\text{Theoretical FLOPs per Token} \times \text{Throughput [tokens/s]}}{\text{Peak Hardware FLOPs Capability}} Typical decode MFU is 10–30%10\text{--}30\%; prefill MFU reaches 40–60%40\text{--}60\%. 3. Economic Unit Cost: Cost per Million Tokens=Instance Cost per Hour×106Throughput [tokens/hour]\text{Cost per Million Tokens} = \frac{\text{Instance Cost per Hour} \times 10^6}{\text{Throughput [tokens/hour]}} An optimization is only viable if Cost decreases without violating accuracy thresholds.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Amdahl's Law Warning: Before writing custom kernels or quantizing weights, run NCU (NVIDIA Nsight Compute) or PyTorch Profiler. If profiling reveals that 85%85\% of time is spent loading weights in GEMV, optimizing FlashAttention in prefill will yield at most a 15%15\% theoretical speedup. ② Tail Latency (P99) Vulnerability: Average latency is deceiving. Techniques like naive speculative decoding or unchunked prefills can improve average throughput while creating massive P99 latency spikes due to rejection rollbacks and scheduling bubbles. Production SLAs depend on P99, not mean. ③ Task Quality Sanity Checks: Never validate quantization or pruning solely with Perplexity (PPL). Always evaluate task-specific benchmarks: MMLU (knowledge), GSM8K (math), HumanEval (code), and RULER (retrieval), ensuring degradation remains <1%<1\%. ④ Workload Distribution Realism: Synthetic benchmarks using fixed-length prompts (e.g., 512 in / 512 out) produce skewed results. Real production workloads follow heavy-tailed distributions (Poisson arrivals, variable lengths); benchmarks must simulate real traffic traces via tools like ShareGPT datasets. ⑤ Interview Strategy: Structure your response into the 4-phase framework (Diagnosis →\to Metrics →\to Quality →\to Economics), define Goodput vs Throughput, and explain why P99 tail latency is critical.
⚠️ Common Interview Pitfalls
  • ✕
    Optimizing without profiling (violating Amdahl's Law by optimizing non-bottleneck components)
  • ✕
    Evaluating average latency while ignoring P99 tail latency spikes that violate production SLAs
  • ✕
    Measuring raw throughput instead of Goodput (serving at 2x throughput is worthless if 50% of requests violate latency SLOs)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is Goodput within latency SLOs a far more realistic metric than peak raw throughput for commercial serving?
  • ?
    How do you design an A/B test to verify that an inference optimization caused zero degradation to user satisfaction?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-110: Impact of Batching, Kernel Fusion, and Operator Optimization on Serving Throughput📋 Back to BankNext →M4-112: Comparison of Post-Training Quantization Algorithms: GPTQ, AWQ, and SmoothQuant