M4-101M4: Sequences & TransformersModel Compression & DistillationMedium
⚡Flashcard Drill➔Mastery:

On-Policy vs. Off-Policy Data in Knowledge Distillation for LLMs

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-102 →
📐 Mathematical Definition
off-policy: x∼Dteacher;on-policy: x∼πstudent\text{off-policy}:\ x\sim\mathcal{D}_{\text{teacher}};\qquad \text{on-policy}:\ x\sim\pi_{\text{student}}
⚡ One-Sentence Core Answer
Off-policy distillation trains the student on static teacher-generated text distributions, while on-policy distillation trains the student on its own self-generated responses evaluated by teacher feedback, eliminating distribution shift during autoregressive deployment.

📌 Key Takeaways

  • •
    Off-policy distillation: student learns from pre-generated teacher corpora via standard supervised cross-entropy; fast and stable, but suffers from exposure bias and covariate shift
  • •
    On-policy distillation (SeqKD / MiniLLM): student generates completions from prompts, and teacher provides per-token feedback or reverse KL divergence (DKL(Pstudent∥Pteacher)D_{\text{KL}}(P_{\text{student}} \| P_{\text{teacher}}))
  • •
    Distribution alignment: on-policy distillation forces the student to recover from its own generation mistakes, yielding significantly higher fluency and lower hallucination rates

📐 Mathematical Derivations

Mathematical Mechanism: 1. Off-Policy Distillation (Forward KL / Supervised Learning): Training objective samples sequences yy from teacher distribution PT(y∣x)P_T(y | x): Loff-policy=Ex∼D,y∼PT[−log⁡PS(y∣x)]=Ex[DKL(PT ∥ PS)]\mathcal{L}_{\text{off-policy}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim P_T} \left[ -\log P_S(y | x) \right] = \mathbb{E}_{x} \left[ D_{\text{KL}}(P_T \, \| \, P_S) \right] Forward KL is mode-covering: the student must assign non-zero probability everywhere the teacher places mass. If the student has lower capacity, it spreads probability mass thinly, generating blurry or ungrammatical text. 2. On-Policy Distillation (Reverse KL / Policy Gradient): Training samples sequences yy from the student's current policy PS(y∣x)P_S(y | x): Lon-policy=Ex∼D,y∼PS[DKL(PS ∥ PT)]=∑tPS(yt∣y<t,x)log⁡PS(yt∣y<t,x)PT(yt∣y<t,x)\mathcal{L}_{\text{on-policy}} = \mathbb{E}_{x \sim \mathcal{D}, y \sim P_S} \left[ D_{\text{KL}}(P_S \, \| \, P_T) \right] = \sum_{t} P_S(y_t | y_{<t}, x) \log \frac{P_S(y_t | y_{<t}, x)}{P_T(y_t | y_{<t}, x)} Reverse KL is mode-seeking: the student concentrates its capacity on the major modes of the teacher distribution where it is confident, avoiding generating nonsensical hallucinations in regions it cannot model.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Covariate Shift & Exposure Bias: In off-policy training, the student is conditioned strictly on ground-truth teacher prefixes (y<tteachery_{<t}^{\text{teacher}}). During real-world autoregressive inference, the student conditions on its own prior generations (y<tstudenty_{<t}^{\text{student}}). Any small error at step tt sends the student out-of-distribution, cascading into complete generation collapse. On-policy training exposes the student to its own mistakes during training, teaching it error recovery. ② Computational Complexity: Off-policy training is simple standard gradient descent over static offline datasets. On-policy training requires active online token sampling from the student, followed by teacher forward passes and policy gradient / PPO-like updates, multiplying training compute by 3–5×3\text{--}5\times. ③ Modern Industrial Applications: Training compact reasoning models (e.g., DeepSeek-R1-Distill, Qwen-2.5-Math) starts with off-policy pre-distillation on millions of teacher tokens, followed by on-policy reinforcement learning or on-policy distillation to sharpen output modes. ④ Optimization Stability: On-policy RL-based distillation suffers from high gradient variance; techniques like MiniLLM or Generalized Knowledge Distillation (GKD) stabilize optimization. ⑤ Interview Strategy: Contrast mode-covering (Forward KL) with mode-seeking (Reverse KL), explain exposure bias, and justify when to pay the higher computational cost of on-policy distillation.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming off-policy distillation on teacher text is equivalent to on-policy generation training (ignores covariate shift during autoregression)
  • ✕
    Using Reverse KL on a model without sufficient pre-training (leads to mode collapse into repetitive single phrases)
  • ✕
    Overlooking that on-policy distillation requires generating tokens dynamically during the training loop
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Forward KL lead to mode-covering behavior while Reverse KL leads to mode-seeking behavior?
  • ?
    How does Generalized Knowledge Distillation (GKD) interpolate between on-policy and off-policy data distributions?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-100: Three Granularities of Model Pruning: Unstructured, Semi-Structured 2:4, and Structured📋 Back to BankNext →M4-102: Systematic Categorization of Inference Acceleration Pathways and Trade-offs