🖥️ AI InfrastructureID:
quantization-inferenceInference Quantization (cross-module)
推理量化(跨模块)
🎯Core Definition
Inference quantization = casting model weights (and optionally activations and KV cache) from FP16 to INT8/INT4/FP8 to trade memory for serving throughput (quantization theory and error analysis live in the LLM module; this card focuses on serving-side deployment). Three mainstream schemes: ① INT8 W8A8 (weights and activations both 8-bit): activations have outliers, so SmoothQuant shifts the outliers into the weights before activation quantization; ② INT4 W4A16 (4-bit weights, FP16 activations): AWQ applies activation-aware channel-wise scales into the weights to protect the "important channels" with the largest output impact — small calibration set, no backprop, cheaper than GPTQ; ③ FP8 (E4M3): native Tensor Core support on H100/B200, large dynamic range, near-zero calibration cost, the new serving default. Memory ledger: 7B FP16 ≈ 14GB → INT8 ≈ 7GB → INT4 ≈ 3.5GB; 70B goes from ≈ 140GB to ≈ 35GB, fitting one GPU.
💡Use Cases
squeezing models onto fewer GPUs and packing larger concurrent batches on each card to cut per-token cost; in interviews the LLM module tests principles (quantization error/calibration) while AI_Infra tests deployment — GPTQ/AWQ/FP8 backend selection in vLLM/SGLang, group size (e.g. 128/32), and batch-size vs memory accounting.
⚡Key Problems Solved
vs FP16, INT4 weights use 4× less memory (14GB → 3.5GB), 4× the model scale fits a single GPU, and the freed memory buys larger concurrent batches, roughly doubling serving throughput; the cost is precision loss from activation outliers and quantization noise, mitigated by calibration data and algorithms like AWQ/SmoothQuant — memory, precision and throughput form the core trade-off triangle, settled against latency budgets and quality gates on real eval sets.
🎯5 High-Frequency Exam Points
1
Difference between W8A8 and W4A16? Why is activation quantization harder than weight quantization (outliers) and how does SmoothQuant migrate outliers into the weights?
2
How AWQ works: why do activation-aware scales on "important channels" preserve accuracy? Why is calibration cheaper than GPTQ (no backprop)?
3
FP8 serving: E4M3 layout and native H100 support; vs INT8 it has wider dynamic range and needs less calibration; when would you still pick INT4?
4
Serving math: 7B FP16 ≈ 14GB → INT8 ≈ 7GB → INT4 ≈ 3.5GB; how do batch size and throughput scale in the same memory? How do group sizes 128/32 trade precision and overhead?
5
Memory-precision-throughput trade-off: when to choose post-training quantization (PTQ) vs quantization-aware training (QAT)? How do you verify quantization loss stays within quality gates before rollout?
📖 In-depth Guide:📄 quantization-and-compression →
Updated 2026-08-12🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Inference Quantization (cross-module)"
Single choice pitfall questions with instant feedback and mistake tracking.