Q1Medium★ GPU Profiling, Serving & Kernel FusionPart Q · AI Infra Hardware & Memory Scheduling

GPU Roofline Arithmetic Intensity & Ridge-Point Analyzer

⚡Flashcard Drill➔

Calculate hardware ridge points and classify GEMM shapes as HBM memory-bound vs TensorCore compute-bound from first principles.

⏱️ Time: O(1)
💾 Space: O(1)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
def roofline_gemm_analysis(M: int, N: int, K: int, peak_tflops: float, hbm_bw_tbs: float, dtype_bytes: int = 2) -> dict:
flops =
bytes_moved = float((M * K + K * N + M * N) * dtype_bytes)
intensity = flops / max(bytes_moved, 1.0)
ridge_point = (peak_tflops * 1e12) / (hbm_bw_tbs * 1e12)
bound = "memory_bound" if intensity < ridge_point else "compute_bound"
t_compute_ms = flops / (peak_tflops * 1e12) * 1e3
t_memory_ms = bytes_moved / (hbm_bw_tbs * 1e12) * 1e3
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does weight-only INT4 quantization speed up decode even though dequantization adds extra ALU instructions?
Q2: How does FlashAttention shift attention upward on the Roofline?
Next: Contiguous KV Memory Defragmentation vs PagedAttention →
💡

Core Mental Anchor / Mnemonic

2MNK FLOPs over byte traffic: small M hits memory ceiling, large M hits compute roof

📐 Mathematical Derivation & Core Formula

$$I = \frac{\text{FLOPs}}{\text{Bytes}_{\text{HBM}}},\quad I_{\text{ridge}} = \frac{\text{Peak FLOPS}}{\text{Peak HBM Bandwidth}},\quad T_{\text{ideal}} = \max\left(\frac{\text{FLOPs}}{P_{\text{comp}}},\; \frac{\text{Bytes}}{B_{\text{mem}}}\right)$$
A GEMM of (M,K)×(K,N)(M, K)\times(K, N) performs 2MNK2MNK FLOPs while transferring at least b(MK+KN+MN)b(MK + KN + MN) bytes over HBM. With batch/token M=1M=1, intensity collapses near 11 FLOP/byte—orders of magnitude below H100's ∼295\sim 295 ridge point—making decode fundamentally memory-bandwidth bound.

🔄 Tensor Dimensions & Shape Flow

GEMM (M, N, K) + hardware specs -> intensity, ridge_point, bottleneck bound

🛡️ Numerical Stability & Pitfalls Checklist

  • Match byte widths carefully (2 for FP16/BF16, 1 for FP8/INT8) when turning elements into bytes
  • Use ridge point peak_tflops / hbm_bandwidth to derive the exact crossover intensity
  • Recognize that increasing batch size M or fusing elementwise ops shifts workloads rightward on the roofline

💻 Reference Code Implementation

def roofline_gemm_analysis(M: int, N: int, K: int, peak_tflops: float, hbm_bw_tbs: float, dtype_bytes: int = 2) -> dict:
    flops = 2.0 * M * N * K
    bytes_moved = float((M * K + K * N + M * N) * dtype_bytes)
    intensity = flops / max(bytes_moved, 1.0)
    ridge_point = (peak_tflops * 1e12) / (hbm_bw_tbs * 1e12)
    bound = "memory_bound" if intensity < ridge_point else "compute_bound"
    t_compute_ms = flops / (peak_tflops * 1e12) * 1e3
    t_memory_ms = bytes_moved / (hbm_bw_tbs * 1e12) * 1e3
    return {"intensity": intensity, "ridge_point": ridge_point, "bound": bound, "ideal_ms": max(t_compute_ms, t_memory_ms)}

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

# H100 approx: 989 TFLOPS FP16, 3.35 TB/s HBM -> ridge ~ 295 FLOP/byte
dec = roofline_gemm_analysis(M=1, N=4096, K=4096, peak_tflops=989.0, hbm_bw_tbs=3.35)
pre = roofline_gemm_analysis(M=2048, N=4096, K=4096, peak_tflops=989.0, hbm_bw_tbs=3.35)
assert dec['bound'] == 'memory_bound' and dec['intensity'] < 2.0
assert pre['bound'] == 'compute_bound' and pre['intensity'] > 600.0
print('Q1 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why does weight-only INT4 quantization speed up decode even though dequantization adds extra ALU instructions?
Decode is overwhelmingly HBM-bandwidth bound; cutting weight bytes by 4x speeds up transfer directly while abundant idle ALU cycles absorb dequantization for free.
Q2:How does FlashAttention shift attention upward on the Roofline?
By tiling Q/K/V inside fast on-chip SRAM instead of writing the N x N attention matrix out to HBM.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Platt Scaling & Expected Calibration Error (ECE) for Ads pCTRBack to Part Q & All 105Next: Contiguous KV Memory Defragmentation vs PagedAttention →