Inference Quantization (cross-module)

Inference quantization = casting model weights (and optionally activations and KV cache) from FP16 to INT8/INT4/FP8 to trade memory for serving throughput (quantization theory and error analysis live in the LLM module; this card focuses on serving-side deployment). Three mainstream schemes: ① INT8 W8A8 (weights and activations both 8-bit): activations have outliers, so SmoothQuant shifts the outliers into the weights before activation quantization; ② INT4 W4A16 (4-bit weights, FP16 activations): AWQ applies activation-aware channel-wise scales into the weights to protect the "important channels" with the largest output impact — small calibration set, no backprop, cheaper than GPTQ; ③ FP8 (E4M3): native Tensor Core support on H100/B200, large dynamic range, near-zero calibration cost, the new serving default. Memory ledger: 7B FP16 ≈ 14GB → INT8 ≈ 7GB → INT4 ≈ 3.5GB; 70B goes from ≈ 140GB to ≈ 35GB, fitting one GPU.

squeezing models onto fewer GPUs and packing larger concurrent batches on each card to cut per-token cost; in interviews the LLM module tests principles (quantization error/calibration) while AI_Infra tests deployment — GPTQ/AWQ/FP8 backend selection in vLLM/SGLang, group size (e.g. 128/32), and batch-size vs memory accounting.

vs FP16, INT4 weights use 4× less memory (14GB → 3.5GB), 4× the model scale fits a single GPU, and the freed memory buys larger concurrent batches, roughly doubling serving throughput; the cost is precision loss from activation outliers and quantization noise, mitigated by calibration data and algorithms like AWQ/SmoothQuant — memory, precision and throughput form the core trade-off triangle, settled against latency budgets and quality gates on real eval sets.

Difference between W8A8 and W4A16? Why is activation quantization harder than weight quantization (outliers) and how does SmoothQuant migrate outliers into the weights?

How AWQ works: why do activation-aware scales on "important channels" preserve accuracy? Why is calibration cheaper than GPTQ (no backprop)?

FP8 serving: E4M3 layout and native H100 support; vs INT8 it has wider dynamic range and needs less calibration; when would you still pick INT4?

Serving math: 7B FP16 ≈ 14GB → INT8 ≈ 7GB → INT4 ≈ 3.5GB; how do batch size and throughput scale in the same memory? How do group sizes 128/32 trade precision and overhead?

Memory-precision-throughput trade-off: when to choose post-training quantization (PTQ) vs quantization-aware training (QAT)? How do you verify quantization loss stays within quality gates before rollout?