Low-Bit INT8/INT4

Low-bit quantization drops weights/activations to 8-bit or 4-bit integers. Storage goes from 2 bytes (FP16) to 1 byte (INT8) to 0.5 byte (INT4) per weight — INT4 weights cut weight memory by 75% (FP16 70B needs 140GB, INT4 only ~35GB). Each tensor is aligned with a scale $s$ and zero-point $z$: $x_q = \text{round}(x/s) + z$; 4-bit typically uses group-wise quantization (e.g., one shared scale per 128 weights) to avoid small-scale channels being swallowed by a wide range.

memory-constrained deployment (7B-70B on consumer GPUs and edge devices), inference bandwidth optimization, KV cache quantization, and offline one-shot conversion pipelines.

bigger models mean costlier memory and bandwidth; INT4 removes 75% of weight memory with no added latency, and calibration-set choice plus group-wise quantization keep accuracy loss manageable — the calibration set is the key to precision, since distribution mismatch is the main killer at low bits.

Test Your Knowledge: Practice Questions for "Low-Bit INT8/INT4"

Single choice pitfall questions with instant feedback and mistake tracking.