M4-099M4: Sequences & TransformersModel Compression & DistillationEasy
⚡Flashcard Drill➔Mastery:
Knowledge Distillation Framework and the Role of Temperature $T$
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Knowledge distillation transfers representation knowledge from a large teacher model to a compact student model by matching soft probability distributions, with temperature controlling the smoothness and information density of the dark knowledge dark probabilities.
📌 Key Takeaways
- •Distillation objective: convex combination of hard cross-entropy loss against ground-truth labels and soft KL-divergence loss against teacher predictions
- •Dark knowledge: non-target output probabilities carry rich inter-class similarity structures (e.g., a truck is closer to a car than to a bird) that hard labels discard
- •Role of temperature : scales logits before softmax; higher softens the distribution, revealing fine-grained relative probabilities among non-target classes
📐 Mathematical Derivations
Mathematical Mechanism: Given teacher logits and student logits : 1. Softened Probability Distribution: For temperature , the soft probability for class is: When , collapses into a one-hot vector (hard labels). When , approaches a uniform distribution . For moderate (e.g., ), relative magnitude differences between negative classes are amplified. 2. Total Loss Formulation: 3. The $T^2$ Scaling Factor Rationale: In the high-temperature limit (), using Taylor expansion : The gradient scales proportionally to . Multiplying the soft loss by ensures that the gradient magnitude from the teacher remains balanced with the hard label cross-entropy loss regardless of the temperature chosen.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Temperature Tuning Strategy: If is too low (), only the top-1 correct class has non-zero probability, making distillation redundant with hard label training. If is too high (), probabilities degrade into uninformative uniform noise. Optimal range is typically for classification and for language models. ② Feature-Based & Intermediate Distillation: Beyond output logits, modern distillation matches hidden representations (FitNets) or attention score distributions (), providing direct supervision to intermediate layers (MiniLM). ③ Model Capacity Gap: If the teacher is vastly larger than the student (e.g., 70B teacher to 1B student), the student cannot fit the teacher's complex distribution, leading to student underfitting. Staged distillation via an intermediate assistant model (70B 14B 1B) yields higher student performance. ④ Distillation vs Direct Pre-training: Training on teacher-generated synthetic text (data distillation) often outperforms direct logit distillation for LLMs due to vocabulary size differences and generation divergence. ⑤ Interview Strategy: Write the softened softmax equation, derive the gradient in the high-temperature limit to explain why the multiplier is mathematically required, and define 'dark knowledge'.
⚠️ Common Interview Pitfalls
- ✕Forgetting to multiply the distillation loss by when using temperature (causes student gradients from soft targets to vanish)
- ✕Setting temperature excessively high, which flattens probabilities into uniform noise and destroys semantic class relationships
- ✕Assuming distillation can only be applied to output classification logits rather than intermediate attention/hidden states
🎯 Interviewer Follow-up Inquiries
- ?Why is the factor mathematically essential when computing gradients for the soft distillation loss?
- ?How does intermediate feature matching (like MiniLM attention transfer) improve distillation over pure logit matching?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚