M4-065M4: Sequences & TransformersKV Cache & Inference OptimizationsHard
⚡Flashcard Drill➔Mastery:

Throughput vs. Latency Trade-offs in Inference: TTFT, TPOT, and Batch Size

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-066 →
📐 Mathematical Definition
throughput∝B;TPOT≈const+contention;maximize TP under SLO\text{throughput}\propto B;\qquad \text{TPOT}\approx\text{const}+\text{contention};\qquad \text{maximize TP under SLO}
⚡ One-Sentence Core Answer
LLM inference balances Time-to-First-Token (TTFT) and Time-Per-Output-Token (TPOT) against global system throughput via batch size, navigating the Pareto frontier between compute saturation and memory bandwidth constraints.

📌 Key Takeaways

  • •
    TTFT (prefill latency): user-perceived responsiveness; compute-bound and sensitive to prompt length, input token caching, and GPU FLOPs
  • •
    TPOT (decode latency): user-perceived reading fluency; memory bandwidth-bound, scaling with model size, KV cache access time, and batch size
  • •
    Throughput (tokens/sec/GPU): scales monotonically with batch size until VRAM or compute limits are reached, but directly degrades TPOT

📐 Mathematical Derivations

Mathematical Mechanism: 1. TTFT Formulation: For prompt length LpromptL_{\text{prompt}} and batch BprefillB_{\text{prefill}}: TTFT≈2P⋅LpromptFLOPshardware⋅MFU+Tqueue\text{TTFT} \approx \frac{2 P \cdot L_{\text{prompt}}}{\text{FLOPs}_{\text{hardware}} \cdot \text{MFU}} + T_{\text{queue}} Where PP is parameter count, and MFU\text{MFU} is Model FLOPs Utilization (~40-60% during prefill). 2. TPOT Formulation: During decoding with batch size BdecodeB_{\text{decode}}, generating one token requires loading all model weights WW and the KV caches for all BdecodeB_{\text{decode}} requests: TPOT(B)≈Bytes(W)+∑i=1B2NHKVdkLibMemory BandwidthHBM+2P⋅BFLOPshardware\text{TPOT}(B) \approx \frac{\text{Bytes}(W) + \sum_{i=1}^B 2 N H_{\text{KV}} d_k L_i b}{\text{Memory Bandwidth}_{\text{HBM}}} + \frac{2 P \cdot B}{\text{FLOPs}_{\text{hardware}}} When BB is small (e.g., B≤8B \le 8), the memory weight loading term Bytes(W)Bandwidth\frac{\text{Bytes}(W)}{\text{Bandwidth}} dominates, making TPOT nearly constant with respect to BB. As BB increases, the compute term and KV cache bandwidth become significant, causing TPOT to rise linearly. 3. System Throughput: Throughput(B)=BTPOT(B)[tokens / second]\text{Throughput}(B) = \frac{B}{\text{TPOT}(B)} \quad \text{[tokens / second]} Because TPOT(B)\text{TPOT}(B) is flat for small BB, increasing BB initially yields a linear increase in throughput at almost zero TPOT penalty. Beyond the saturation point B∗B^*, throughput plateaus while TPOT degrades linearly.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Knee Point ($B^*$): The optimal operating point on the Roofline model occurs when arithmetic intensity 2PBBytes(W)\frac{2 P B}{\text{Bytes}(W)} matches the hardware balance point TFLOPsTB/s\frac{\text{TFLOPs}}{\text{TB/s}}. For A100 (~150 FLOPs/byte), B∗≈75–100B^* \approx 75\text{--}100. Beyond this point, decoding transitions from bandwidth-bound to compute-bound. ② SLA Trade-offs: Interactive chat requires tight TPOT (<30–50 ms/token<30\text{--}50\text{ ms/token}, matching human reading speed of ~20-30 tokens/sec), requiring conservative batch sizes (B≤32B \le 32). Offline batch processing (e.g., synthetic data generation, summarization) targets maximum throughput (B≥128–256B \ge 128\text{--}256), tolerating TPOT of 100–200 ms100\text{--}200\text{ ms}. ③ Tensor Parallelism vs Batching: Increasing TP (e.g., TP=2 to TP=8) reduces per-GPU weight size and divides latency by ≈NGPU\approx N_{\text{GPU}}, improving TPOT for latency-critical SLAs, but incurs all-reduce communication overhead. ④ Queue Delay Dynamics: If traffic spikes and arrivals exceed serving capacity, queue delay TqueueT_{\text{queue}} explodes exponentially, dominating TTFT. ⑤ Interview Strategy: Plot the Roofline curve of TPOT vs. Batch Size, identify the knee point where throughput saturates, and define SLA-driven parameter tuning for interactive vs. batch workloads.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming increasing batch size always increases latency linearly (at small batch sizes, TPOT is virtually flat due to memory bandwidth limits)
  • ✕
    Optimizing exclusively for throughput while violating user-facing TPOT SLAs (>80 ms feels sluggish)
  • ✕
    Ignoring the impact of queueing delay on TTFT under high load
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you determine the optimal batch size threshold B∗B^* using the Roofline model for an 8x H100 node?
  • ?
    What strategies allow serving engines to maintain low TPOT while handling high-throughput background workloads?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-064: Speculative Decoding Variants: Medusa, EAGLE, and Multi-Token Prediction📋 Back to BankNext →M4-066: Prefill-Decode Disaggregation Architecture