GPU Hardware & HBM

The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB shared-memory SRAM per SM). The memory hierarchy, fastest to slowest: registers (256KB per SM) → shared memory SRAM → L2 (50MB) → HBM (80GB) → host DRAM/NVMe; each level drops roughly an order of magnitude in bandwidth while gaining capacity. Memory wall: a 70B model in FP16 is 140GB of weights; reading them once at 3.35TB/s takes $\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}$, while one token of forward compute is roughly $2 \times 70\text{B} = 140\,\text{GFLOP}$, needing only $\approx 0.14\,\text{ms}$ at 989 TFLOPS — a ~300:1 ratio. Since weights live in HBM and every use re-reads all of them, training/inference is locked to the bandwidth wall (bandwidth-bound). Arithmetic intensity $I = \frac{\text{FLOP}}{\text{byte}}$ decides the regime: H100's balance point is $\frac{989\,\text{TFLOP/s}}{3.35\,\text{TB/s}} \approx 295\,\text{FLOP/byte}$; ops below it are bandwidth-bound (LayerNorm, activations, elementwise), above it compute-bound (large matmuls).

the first lesson of training-infra interviews — memory estimation (16Ψ, activations, KV Cache all reference the 80GB HBM), operator bottleneck analysis (torch.profiler: memory-bound vs compute-bound), and parallelism choice (TP needs intra-node NVLink for its frequent small messages of size ~$b \times s \times h$).

explains why large-model training/inference never saturates FLOPs — the 140GB of weights must be re-read from HBM constantly, 42ms of transfer vs 0.14ms of compute (~300×). Every infra optimization (FlashAttention's SRAM reuse, quantization shrinking bytes, TP/PP parallel amortization, KV Cache tuning) is ultimately a fight against this bandwidth wall.

Recite H100 specs: 80GB HBM3, 3.35TB/s bandwidth, 989 TFLOPS FP16, 50MB L2; compute the balance arithmetic intensity $\approx 295\,\text{FLOP/byte}$ and classify the GPU as compute- or bandwidth-bound.

Hand-derive the memory wall: 70B FP16 weights = 140GB, one pass costs $\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}$; one token's forward is $2 \times 70\text{B}$ FLOP = $\approx 0.14\,\text{ms}$ at 989 TFLOPS; ~300:1 ratio, explaining why training is bandwidth-bound.

Sketch the memory hierarchy: registers → SRAM/shared memory → L2 → HBM → host memory with typical capacity/bandwidth orders (H100: 228KB/SM, 50MB, 80GB); why FlashAttention keeps intermediates in SRAM instead of writing back to HBM.

Use arithmetic intensity $I=\text{FLOP/byte}$ to classify ops: state H100's balance point $\approx 295$; give bandwidth-bound examples (LayerNorm/activations/GELU) vs compute-bound (large matmul) and explain the reasoning.

Argue from bandwidth/capacity: why 70B cannot fit one card (140GB weights > 80GB HBM), requiring weight sharding (TP/ZeRO) or quantization; why byte-reduction (quantization, recompute) beats raw FLOPs when weights are re-read wholesale.