M4-110M4: Sequences & TransformersQuantization & AccelerationHard
⚡Flashcard Drill➔Mastery:
Impact of Batching, Kernel Fusion, and Operator Optimization on Serving Throughput
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Serving throughput is maximized by batching to amortize weight reading over multiple requests, kernel fusion to eliminate redundant HBM memory roundtrips, and operator optimization to maximize hardware Tensor Core utilization.
📌 Key Takeaways
- •Batching (continuous batching): amortizes the fixed memory cost of loading billions of model weight parameters over multiple concurrent requests, turning GEMV into compute-dense GEMM
- •Kernel fusion: fuses consecutive element-wise operations (e.g., LayerNorm + Residual + Linear or SwiGLU) into a single GPU kernel, eliminating intermediate HBM read/write traffic
- •Operator optimization (FlashAttention / FlashDecoding): restructures memory access patterns around on-chip SRAM tiling to bypass the GPU memory wall
📐 Mathematical Derivations
Mathematical Mechanism: 1. Weight Amortization via Batching: In decoding, loading model weights ( for 7B FP16) from HBM is a fixed cost per step. The arithmetic intensity is: At , FLOP/byte (memory-bound). At , FLOPs/byte, approaching GPU saturation. Batching increases throughput linearly until compute saturation. 2. Kernel Fusion Memory Traffic Elimination: Consider an unfused sequence: Residual Add RMSNorm GEMM: - Unfused: 1) Read , read residual, write to HBM; 2) Read from HBM, compute RMSNorm, write to HBM; 3) Read from HBM, compute GEMM. Total Memory IO: bytes. - Fused: Load and residual into SRAM once, compute addition and RMSNorm in registers, directly pass activations to GEMM. Total Memory IO: bytes ( less memory traffic!).
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Roofline Optimization Hierarchy: - Decoding is memory bandwidth-bound: Batching is the #1 optimization because it directly amortizes the dominant memory bottleneck. - Element-wise operations are bandwidth-bound: Kernel fusion is the #2 optimization because it slashes intermediate tensor read/writes. - Attention is memory access-bound: FlashAttention/FlashDecoding is the #3 optimization because SRAM tiling bypasses HBM. ② Latency vs Throughput Trade-off: Pushing batch size to extremes () maximizes cluster throughput (tokens/sec/dollar), but inflates individual request latency (TPOT) and risks out-of-memory preemption. ③ Triton & CUTLASS in Production: Writing custom fused kernels historically required months of low-level CUDA; modern frameworks use OpenAI Triton or CUTLASS templates to generate optimized fused kernels (e.g., Fused SwiGLU, Fused RoPE) with minimal engineering overhead. ④ FlashDecoding for Long Context: FlashAttention parallelizes across query sequence length (). In decode where , FlashAttention underutilizes GPU multiprocessors. FlashDecoding introduces a split-KV dimension, partitioning the long KV history across SMs and reducing final results via parallel tree reduction. ⑤ Interview Strategy: Derive the arithmetic intensity scaling equation for batch size, draw the memory traffic diagram comparing fused vs unfused operators, and explain how FlashDecoding adapts FlashAttention for single-token decoding.
⚠️ Common Interview Pitfalls
- ✕Overlooking that batching is the single most effective lever for increasing decode throughput
- ✕Writing individual CUDA kernels for consecutive element-wise operations without fusion (incurs severe memory bandwidth stalls)
- ✕Using standard FlashAttention during single-token decoding instead of FlashDecoding (causes severe GPU multiprocessor underutilization)
🎯 Interviewer Follow-up Inquiries
- ?How does FlashDecoding parallelize across the KV cache dimension to accelerate long-context single-token decoding?
- ?Why does fusing SwiGLU activation with the gating projection reduce GPU memory traffic?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚