Knowledge Distillation

Knowledge distillation (KD) has a student model mimic the teacher's soft output distribution $p_T = \text{softmax}(z/T)$, where temperature $T$ softens probabilities to expose inter-class dark knowledge; loss is $\mathcal{L} = \alpha \mathcal{L}_{CE}(y, p_S) + (1-\alpha) \cdot T^2 \cdot \text{KL}(p_S \| p_T)$, with $T^2$ compensating the softened gradient scale.

compressing a large model into a deployable small one, transferring knowledge on unlabeled data (teacher outputs as soft labels), and cross-architecture learning (LLM teacher to a smaller transformer); in LLM compression it is orthogonal to quantization — quantization cuts bit-width, distillation cuts model capacity, and the two stack.

when models are too large to deploy, distillation distills the teacher's generalization into the student at lower training cost than training from scratch, and soft labels retain inter-class similarity information that hard labels drop; together with quantization it forms two complementary compression axes.

Why learn from soft targets instead of hard labels?

Role of temperature T; effects of T too large or too small?

How do distillation and quantization relate — why are they orthogonal?

Common KD variants (logits, features, self-distillation)?

What happens when the student-teacher capacity gap is large, and how to mitigate?