Q1Medium★ GPU Profiling, Serving & Kernel FusionPart Q · AI Infra Hardware & Memory Scheduling
GPU Roofline Arithmetic Intensity & Ridge-Point Analyzer
Calculate hardware ridge points and classify GEMM shapes as HBM memory-bound vs TensorCore compute-bound from first principles.
⏱️ Time:
O(1)💾 Space:
O(1)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def roofline_gemm_analysis(M: int, N: int, K: int, peak_tflops: float, hbm_bw_tbs: float, dtype_bytes: int = 2) -> dict:
flops =
bytes_moved = float((M * K + K * N + M * N) * dtype_bytes)
intensity = flops / max(bytes_moved, 1.0)
ridge_point = (peak_tflops * 1e12) / (hbm_bw_tbs * 1e12)
bound = "memory_bound" if intensity < ridge_point else "compute_bound"
t_compute_ms = flops / (peak_tflops * 1e12) * 1e3
t_memory_ms = bytes_moved / (hbm_bw_tbs * 1e12) * 1e3
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does weight-only INT4 quantization speed up decode even though dequantization adds extra ALU instructions?
Q2: How does FlashAttention shift attention upward on the Roofline?
💡
Core Mental Anchor / Mnemonic
2MNK FLOPs over byte traffic: small M hits memory ceiling, large M hits compute roof
📐 Mathematical Derivation & Core Formula
$$I = \frac{\text{FLOPs}}{\text{Bytes}_{\text{HBM}}},\quad I_{\text{ridge}} = \frac{\text{Peak FLOPS}}{\text{Peak HBM Bandwidth}},\quad T_{\text{ideal}} = \max\left(\frac{\text{FLOPs}}{P_{\text{comp}}},\; \frac{\text{Bytes}}{B_{\text{mem}}}\right)$$
A GEMM of performs FLOPs while transferring at least bytes over HBM. With batch/token , intensity collapses near FLOP/byte—orders of magnitude below H100's ridge point—making decode fundamentally memory-bandwidth bound.
🔄 Tensor Dimensions & Shape Flow
GEMM (M, N, K) + hardware specs -> intensity, ridge_point, bottleneck bound
🛡️ Numerical Stability & Pitfalls Checklist
- Match byte widths carefully (2 for FP16/BF16, 1 for FP8/INT8) when turning elements into bytes
- Use ridge point peak_tflops / hbm_bandwidth to derive the exact crossover intensity
- Recognize that increasing batch size M or fusing elementwise ops shifts workloads rightward on the roofline
💻 Reference Code Implementation
def roofline_gemm_analysis(M: int, N: int, K: int, peak_tflops: float, hbm_bw_tbs: float, dtype_bytes: int = 2) -> dict:
flops = 2.0 * M * N * K
bytes_moved = float((M * K + K * N + M * N) * dtype_bytes)
intensity = flops / max(bytes_moved, 1.0)
ridge_point = (peak_tflops * 1e12) / (hbm_bw_tbs * 1e12)
bound = "memory_bound" if intensity < ridge_point else "compute_bound"
t_compute_ms = flops / (peak_tflops * 1e12) * 1e3
t_memory_ms = bytes_moved / (hbm_bw_tbs * 1e12) * 1e3
return {"intensity": intensity, "ridge_point": ridge_point, "bound": bound, "ideal_ms": max(t_compute_ms, t_memory_ms)}🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
# H100 approx: 989 TFLOPS FP16, 3.35 TB/s HBM -> ridge ~ 295 FLOP/byte
dec = roofline_gemm_analysis(M=1, N=4096, K=4096, peak_tflops=989.0, hbm_bw_tbs=3.35)
pre = roofline_gemm_analysis(M=2048, N=4096, K=4096, peak_tflops=989.0, hbm_bw_tbs=3.35)
assert dec['bound'] == 'memory_bound' and dec['intensity'] < 2.0
assert pre['bound'] == 'compute_bound' and pre['intensity'] > 600.0
print('Q1 PASS')🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why does weight-only INT4 quantization speed up decode even though dequantization adds extra ALU instructions?
Decode is overwhelmingly HBM-bandwidth bound; cutting weight bytes by 4x speeds up transfer directly while abundant idle ALU cycles absorb dequantization for free.
Q2:How does FlashAttention shift attention upward on the Roofline?
By tiling Q/K/V inside fast on-chip SRAM instead of writing the N x N attention matrix out to HBM.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.