🖥️ AI InfrastructureID: gpu-hardware

GPU Hardware & HBM

Q: Recite H100 specs: 80GB HBM3, 3.35TB/s bandwidth, 989 TFLOPS FP16, 50MB L2; compute the balance arithmetic intensity $\approx 295\,\text{FLOP/byte}$ and classify the GPU as compute- or bandwidth-bound.

【GPU Hardware & HBM 核心考点 1】Recite H100 specs: 80GB HBM3, 3.35TB/s bandwidth, 989 TFLOPS FP16, 50MB L2; compute the balance arithmetic intensity $\approx 295\,\text{FLOP/byte}$ and classify the GPU as compute- or bandwidth-bound.。技术解析：Definition: The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/gpu-hardware

Q: Hand-derive the memory wall: 70B FP16 weights = 140GB, one pass costs $\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}$; one token's forward is $2 \times 70\text{B}$ FLOP = $\approx 0.14\,\text{ms}$ at 989 TFLOPS; ~300:1 ratio, explaining why training is bandwidth-bound.

【GPU Hardware & HBM 核心考点 2】Hand-derive the memory wall: 70B FP16 weights = 140GB, one pass costs $\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}$; one token's forward is $2 \times 70\text{B}$ FLOP = $\approx 0.14\,\text{ms}$ at 989 TFLOPS; ~300:1 ratio, explaining why training is bandwidth-bound.。技术解析：Definition: The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/gpu-hardware

Q: Sketch the memory hierarchy: registers → SRAM/shared memory → L2 → HBM → host memory with typical capacity/bandwidth orders (H100: 228KB/SM, 50MB, 80GB); why FlashAttention keeps intermediates in SRAM instead of writing back to HBM.

【GPU Hardware & HBM 核心考点 3】Sketch the memory hierarchy: registers → SRAM/shared memory → L2 → HBM → host memory with typical capacity/bandwidth orders (H100: 228KB/SM, 50MB, 80GB); why FlashAttention keeps intermediates in SRAM instead of writing back to HBM.。技术解析：Definition: The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/gpu-hardware

Q: Use arithmetic intensity $I=\text{FLOP/byte}$ to classify ops: state H100's balance point $\approx 295$; give bandwidth-bound examples (LayerNorm/activations/GELU) vs compute-bound (large matmul) and explain the reasoning.

【GPU Hardware & HBM 核心考点 4】Use arithmetic intensity $I=\text{FLOP/byte}$ to classify ops: state H100's balance point $\approx 295$; give bandwidth-bound examples (LayerNorm/activations/GELU) vs compute-bound (large matmul) and explain the reasoning.。技术解析：Definition: The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/gpu-hardware

Q: Argue from bandwidth/capacity: why 70B cannot fit one card (140GB weights > 80GB HBM), requiring weight sharding (TP/ZeRO) or quantization; why byte-reduction (quantization, recompute) beats raw FLOPs when weights are re-read wholesale.

【GPU Hardware & HBM 核心考点 5】Argue from bandwidth/capacity: why 70B cannot fit one card (140GB weights > 80GB HBM), requiring weight sharding (TP/ZeRO) or quantization; why byte-reduction (quantization, recompute) beats raw FLOPs when weights are re-read wholesale.。技术解析：Definition: The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/gpu-hardware

GPU 硬件与 HBM

🎯Core Definition

The hardware foundation of GPU training/inference is a three-axis trade-off: compute, memory bandwidth, and memory capacity. Take H100 SXM as the baseline: 80GB HBM3 at 3.35TB/s; 989 TFLOPS dense FP16/BF16 (1979 with sparsity), 1979 TFLOPS FP8; 50MB L2, 132 SMs (228KB shared-memory SRAM per SM). The memory hierarchy, fastest to slowest: registers (256KB per SM) → shared memory SRAM → L2 (50MB) → HBM (80GB) → host DRAM/NVMe; each level drops roughly an order of magnitude in bandwidth while gaining capacity. Memory wall: a 70B model in FP16 is 140GB of weights; reading them once at 3.35TB/s takes

\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}

, while one token of forward compute is roughly

2 \times 70\text{B} = 140\,\text{GFLOP}

, needing only

\approx 0.14\,\text{ms}

at 989 TFLOPS — a ~300:1 ratio. Since weights live in HBM and every use re-reads all of them, training/inference is locked to the bandwidth wall (bandwidth-bound). Arithmetic intensity

I = \frac{\text{FLOP}}{\text{byte}}

decides the regime: H100's balance point is

\frac{989\,\text{TFLOP/s}}{3.35\,\text{TB/s}} \approx 295\,\text{FLOP/byte}

; ops below it are bandwidth-bound (LayerNorm, activations, elementwise), above it compute-bound (large matmuls).

💡Use Cases

the first lesson of training-infra interviews — memory estimation (16Ψ, activations, KV Cache all reference the 80GB HBM), operator bottleneck analysis (torch.profiler: memory-bound vs compute-bound), and parallelism choice (TP needs intra-node NVLink for its frequent small messages of size ~

b \times s \times h

⚡Key Problems Solved

explains why large-model training/inference never saturates FLOPs — the 140GB of weights must be re-read from HBM constantly, 42ms of transfer vs 0.14ms of compute (~300×). Every infra optimization (FlashAttention's SRAM reuse, quantization shrinking bytes, TP/PP parallel amortization, KV Cache tuning) is ultimately a fight against this bandwidth wall.

🎯5 High-Frequency Exam Points

Recite H100 specs: 80GB HBM3, 3.35TB/s bandwidth, 989 TFLOPS FP16, 50MB L2; compute the balance arithmetic intensity

\approx 295\,\text{FLOP/byte}

and classify the GPU as compute- or bandwidth-bound.

Hand-derive the memory wall: 70B FP16 weights = 140GB, one pass costs

\frac{140\,\text{GB}}{3.35\,\text{TB/s}} \approx 42\,\text{ms}

; one token's forward is

2 \times 70\text{B}

FLOP =

\approx 0.14\,\text{ms}

at 989 TFLOPS; ~300:1 ratio, explaining why training is bandwidth-bound.

Sketch the memory hierarchy: registers → SRAM/shared memory → L2 → HBM → host memory with typical capacity/bandwidth orders (H100: 228KB/SM, 50MB, 80GB); why FlashAttention keeps intermediates in SRAM instead of writing back to HBM.

Use arithmetic intensity

I=\text{FLOP/byte}

to classify ops: state H100's balance point

\approx 295

; give bandwidth-bound examples (LayerNorm/activations/GELU) vs compute-bound (large matmul) and explain the reasoning.

Argue from bandwidth/capacity: why 70B cannot fit one card (140GB weights > 80GB HBM), requiring weight sharding (TP/ZeRO) or quantization; why byte-reduction (quantization, recompute) beats raw FLOPs when weights are re-read wholesale.

📖 In-depth Guide:📄 gpu-hardware-and-hbm →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "GPU Hardware & HBM"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardData Pipelines & Streaming Next Card →Collective Comm & NVLink Topology

🔗 More AI Infrastructure Knowledge Cards

Activation Memory Estimation Agent Runtime (cross-module)Autoscaling & Cost Checkpointing & Recovery