M4-110M4: Sequences & TransformersQuantization & AccelerationHard
⚡Flashcard Drill➔Mastery:

Impact of Batching, Kernel Fusion, and Operator Optimization on Serving Throughput

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-111 →
📐 Mathematical Definition
throughput↑ via {batch, fusion, optimized kernels}\text{throughput}\uparrow\ \text{via}\ \{\text{batch},\ \text{fusion},\ \text{optimized kernels}\}
⚡ One-Sentence Core Answer
Serving throughput is maximized by batching to amortize weight reading over multiple requests, kernel fusion to eliminate redundant HBM memory roundtrips, and operator optimization to maximize hardware Tensor Core utilization.

📌 Key Takeaways

  • •
    Batching (continuous batching): amortizes the fixed memory cost of loading billions of model weight parameters over multiple concurrent requests, turning GEMV into compute-dense GEMM
  • •
    Kernel fusion: fuses consecutive element-wise operations (e.g., LayerNorm + Residual + Linear or SwiGLU) into a single GPU kernel, eliminating intermediate HBM read/write traffic
  • •
    Operator optimization (FlashAttention / FlashDecoding): restructures memory access patterns around on-chip SRAM tiling to bypass the GPU memory wall

📐 Mathematical Derivations

Mathematical Mechanism: 1. Weight Amortization via Batching: In decoding, loading model weights WW (14 GB14\text{ GB} for 7B FP16) from HBM is a fixed cost per step. The arithmetic intensity II is: I(B)=2P⋅BBytes(W)+BytesKV(B)≈2P⋅BBytes(W)[for moderate B]I(B) = \frac{2 P \cdot B}{\text{Bytes}(W) + \text{Bytes}_{\text{KV}}(B)} \approx \frac{2 P \cdot B}{\text{Bytes}(W)} \quad [\text{for moderate } B] At B=1B=1, I≈1I \approx 1 FLOP/byte (memory-bound). At B=64B=64, I≈64I \approx 64 FLOPs/byte, approaching GPU saturation. Batching increases throughput linearly until compute saturation. 2. Kernel Fusion Memory Traffic Elimination: Consider an unfused sequence: Residual Add →\to RMSNorm →\to GEMM: - Unfused: 1) Read xx, read residual, write y1y_1 to HBM; 2) Read y1y_1 from HBM, compute RMSNorm, write y2y_2 to HBM; 3) Read y2y_2 from HBM, compute GEMM. Total Memory IO: 7×(B⋅L⋅d)×b7 \times (B \cdot L \cdot d) \times b bytes. - Fused: Load xx and residual into SRAM once, compute addition and RMSNorm in registers, directly pass activations to GEMM. Total Memory IO: 2×(B⋅L⋅d)×b2 \times (B \cdot L \cdot d) \times b bytes (3.5×3.5\times less memory traffic!).

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Roofline Optimization Hierarchy: - Decoding is memory bandwidth-bound: Batching is the #1 optimization because it directly amortizes the dominant memory bottleneck. - Element-wise operations are bandwidth-bound: Kernel fusion is the #2 optimization because it slashes intermediate tensor read/writes. - Attention is memory access-bound: FlashAttention/FlashDecoding is the #3 optimization because SRAM tiling bypasses HBM. ② Latency vs Throughput Trade-off: Pushing batch size to extremes (B=256B=256) maximizes cluster throughput (tokens/sec/dollar), but inflates individual request latency (TPOT) and risks out-of-memory preemption. ③ Triton & CUTLASS in Production: Writing custom fused kernels historically required months of low-level CUDA; modern frameworks use OpenAI Triton or CUTLASS templates to generate optimized fused kernels (e.g., Fused SwiGLU, Fused RoPE) with minimal engineering overhead. ④ FlashDecoding for Long Context: FlashAttention parallelizes across query sequence length (LQL_Q). In decode where LQ=1L_Q = 1, FlashAttention underutilizes GPU multiprocessors. FlashDecoding introduces a split-KV dimension, partitioning the long KV history across SMs and reducing final results via parallel tree reduction. ⑤ Interview Strategy: Derive the arithmetic intensity scaling equation for batch size, draw the memory traffic diagram comparing fused vs unfused operators, and explain how FlashDecoding adapts FlashAttention for single-token decoding.
⚠️ Common Interview Pitfalls
  • ✕
    Overlooking that batching is the single most effective lever for increasing decode throughput
  • ✕
    Writing individual CUDA kernels for consecutive element-wise operations without fusion (incurs severe memory bandwidth stalls)
  • ✕
    Using standard FlashAttention during single-token decoding instead of FlashDecoding (causes severe GPU multiprocessor underutilization)
🎯 Interviewer Follow-up Inquiries
  • ?
    How does FlashDecoding parallelize across the KV cache dimension to accelerate long-context single-token decoding?
  • ?
    Why does fusing SwiGLU activation with the gating projection reduce GPU memory traffic?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-109: Benefits, Risks, and Outlier Sensitivity in KV Cache Quantization📋 Back to BankNext →M4-111: Systematic Metric Framework for Evaluating Inference Optimizations: TTFT, TPOT, MFU, and Cost