M4-065M4: Sequences & TransformersKV Cache & Inference OptimizationsHard
⚡Flashcard Drill➔Mastery:
Throughput vs. Latency Trade-offs in Inference: TTFT, TPOT, and Batch Size
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
LLM inference balances Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) against global system throughput via batch size, navigating the Pareto frontier between compute saturation and memory bandwidth constraints.
📌 Key Takeaways
- •TTFT (prefill latency): user-perceived responsiveness; compute-bound and sensitive to prompt length, input token caching, and GPU FLOPs
- •TPOT (decode latency): user-perceived reading fluency; memory bandwidth-bound, scaling with model size, KV cache access time, and batch size
- •Throughput (tokens/sec/GPU): scales monotonically with batch size until VRAM or compute limits are reached, but directly degrades TPOT
📐 Mathematical Derivations
Mathematical Mechanism: 1. TTFT Formulation: For prompt length and batch : Where is parameter count, and is Model FLOPs Utilization (~40-60% during prefill). 2. TPOT Formulation: During decoding with batch size , generating one token requires loading all model weights and the KV caches for all requests: When is small (e.g., ), the memory weight loading term dominates, making TPOT nearly constant with respect to . As increases, the compute term and KV cache bandwidth become significant, causing TPOT to rise linearly. 3. System Throughput: Because is flat for small , increasing initially yields a linear increase in throughput at almost zero TPOT penalty. Beyond the saturation point , throughput plateaus while TPOT degrades linearly.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Knee Point ($B^*$): The optimal operating point on the Roofline model occurs when arithmetic intensity matches the hardware balance point . For A100 (~150 FLOPs/byte), . Beyond this point, decoding transitions from bandwidth-bound to compute-bound. ② SLA Trade-offs: Interactive chat requires tight TPOT (, matching human reading speed of ~20-30 tokens/sec), requiring conservative batch sizes (). Offline batch processing (e.g., synthetic data generation, summarization) targets maximum throughput (), tolerating TPOT of . ③ Tensor Parallelism vs Batching: Increasing TP (e.g., TP=2 to TP=8) reduces per-GPU weight size and divides latency by , improving TPOT for latency-critical SLAs, but incurs all-reduce communication overhead. ④ Queue Delay Dynamics: If traffic spikes and arrivals exceed serving capacity, queue delay explodes exponentially, dominating TTFT. ⑤ Interview Strategy: Plot the Roofline curve of TPOT vs. Batch Size, identify the knee point where throughput saturates, and define SLA-driven parameter tuning for interactive vs. batch workloads.
⚠️ Common Interview Pitfalls
- ✕Assuming increasing batch size always increases latency linearly (at small batch sizes, TPOT is virtually flat due to memory bandwidth limits)
- ✕Optimizing exclusively for throughput while violating user-facing TPOT SLAs (>80 ms feels sluggish)
- ✕Ignoring the impact of queueing delay on TTFT under high load
🎯 Interviewer Follow-up Inquiries
- ?How do you determine the optimal batch size threshold using the Roofline model for an 8x H100 node?
- ?What strategies allow serving engines to maintain low TPOT while handling high-throughput background workloads?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚