Knowledge Distillation

Knowledge distillation (KD) has a student model mimic the teacher's soft output distribution $p_T = \text{softmax}(z/T)$, where temperature $T$ softens probabilities to expose inter-class dark knowledge; loss is $\mathcal{L} = \alpha \mathcal{L}_{CE}(y, p_S) + (1-\alpha) \cdot T^2 \cdot \text{KL}(p_S \| p_T)$, with $T^2$ compensating the softened gradient scale.

compressing a large model into a deployable small one, transferring knowledge on unlabeled data (teacher outputs as soft labels), and cross-architecture learning (LLM teacher to a smaller transformer); in LLM compression it is orthogonal to quantization — quantization cuts bit-width, distillation cuts model capacity, and the two stack.

when models are too large to deploy, distillation distills the teacher's generalization into the student at lower training cost than training from scratch, and soft labels retain inter-class similarity information that hard labels drop; together with quantization it forms two complementary compression axes.

Test Your Knowledge: Practice Questions for "Knowledge Distillation"

Single choice pitfall questions with instant feedback and mistake tracking.