Model Compression, Pruning & Quant

Model Compression, Pruning & Low-Bit Quantization optimize neural networks for deployment onto edge hardware or high-throughput cloud inference clusters by slashing memory footprints and memory-bandwidth bottlenecks; it encompasses 3 core pillars: 1) Pruning (Structured channel pruning and unstructured magnitude/Hessian weight pruning inducing sparsity); 2) Knowledge Distillation (transferring dark knowledge from large teacher models to compact student nets via temperature-softened logits $p_T = \text{softmax}(z_T / T)$); 3) Uniform Affine Quantization, mapping continuous floats to INT8/INT4/FP8 integers via $x_q = \text{clamp}\left(\text{round}\left(\frac{x}{S}\right) + Z, q_{\min}, q_{\max}\right)$ across Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

Mobile on-device ML deployment (NCNN, CoreML, TFLite), cloud LLM throughput optimization (vLLM, TensorRT-LLM, AWQ), and inference cost reduction.

Heavy multi-billion parameter networks exceed edge device RAM and saturate GPU memory bandwidth in cloud serving; compression and quantization slash memory footprints by 4-8x and triple throughput with negligible accuracy loss.

Derive the mathematical scale factor $S$ and zero-point $Z$ formulas in uniform asymmetric affine quantization?

Explain why QAT utilizes FakeQuantize nodes in the forward pass and Straight-Through Estimators (STE) in the backward pass to bypass non-differentiable rounding?

Why does structured channel/head pruning achieve direct hardware speedups while unstructured sparse matrices require specialized sparse Tensor Cores?

Explain how high distillation temperatures $T$ soften output logit entropy to reveal dark knowledge across non-target classes?

Explain how SmoothQuant mathematically migrates activation quantization difficulty onto weights and how AWQ protects salient weight channels?