M5-025M5: NLP & Large Language ModelsInstruction Tuning & Supervised Fine-TuningHard
⚡Flashcard Drill➔Mastery:

Division of Labor Between SFT and RLHF/DPO

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-026 →
📐 Mathematical Definition
SFT: max⁡log⁡p(y∣x);RLHF/DPO: max⁡ E[r(x,y)] s.t. KL(π∥πref)≤δ\text{SFT}:\ \max\log p(y|x);\qquad \text{RLHF/DPO}:\ \max\ \mathbb{E}[r(x,y)]\ \text{s.t.}\ \text{KL}(\pi\|\pi_{\text{ref}})\le\delta
⚡ One-Sentence Core Answer
SFT establishes foundational instruction following, domain knowledge, and response formatting, whereas RLHF/DPO refines preference boundaries, suppresses low-probability tail errors, and aligns model outputs with human values.

📌 Key Takeaways

  • •
    SFT role: builds the baseline capability to understand user prompts and respond in coherent, structured markdown; unlocks latent pre-training capabilities
  • •
    RLHF / DPO role: explores the solution space, optimizes relative preference nuances, suppresses hallucinations and toxic completions, and resolves edge-case ambiguity
  • •
    Complementary duality: SFT sets the high-quality starting policy πSFT\pi_{\text{SFT}}; RL cannot function without a competent SFT base to explore from

📐 Mathematical Derivations

Mathematical Mechanism: 1. SFT as Maximum Likelihood Estimation (MLE): SFT maximizes data likelihood over positive demonstrations: max⁡θE(x,y)∼Ddemo[log⁡πθ(y∣x)]\max_\theta \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{demo}}} \left[ \log \pi_\theta(y \mid x) \right] MLE treats all demonstration tokens equally. It has no negative feedback mechanism: it cannot explicitly penalize toxic, hallucinated, or sub-optimal answers. 2. RLHF / DPO as Contrastive Optimization: Given prompt xx and pair (yw≻yl)(y_w \succ y_l) (winning vs losing response): max⁡θE(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\max_\theta \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] RLHF/DPO simultaneously pushes up the likelihood of preferred completions while actively pushing down the likelihood of dispreferred completions, carving sharp decision boundaries.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Why RL Cannot Replace SFT: If you initialize RLHF or DPO directly from a raw pre-trained base model, the action space (all vocabulary tokens across length LL) is astronomically vast. The policy generates unformatted, rambling text that receives near-zero reward, resulting in random gradient noise and optimization failure. SFT provides the vital warm-start policy πref\pi_{\text{ref}}. ② Why SFT Cannot Replace RL: In complex tasks, human evaluators can easily pick the better of two answers (yw≻yly_w \succ y_l), but struggle to write a perfect reference answer from scratch. RLHF leverages pairwise comparison data to surpass the demonstration ability of average human annotators. ③ Mode Collapse vs Exploration: SFT suffers from exposure bias (teacher forcing). RL allows the model to sample its own generations autoregressively and receive feedback, learning error correction. ④ The Alignment Tax: Excessive RLHF/DPO can degrade creative writing diversity and multi-step reasoning (the alignment tax), requiring careful KL regularization against πSFT\pi_{\text{SFT}}. ⑤ Interview Strategy: Contrast MLE (positive-only) vs Contrastive RL (positive and negative), explain why SFT is the prerequisite warm-start for RL, and articulate the complementary strengths of both phases.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to run RLHF or DPO directly on a raw base model without SFT warm-start
  • ✕
    Assuming SFT can learn negative constraints effectively (cross-entropy cannot explicitly penalize dispreferred answers without contrastive objectives)
  • ✕
    Pushing RL optimization so aggressively that the model suffers the 'alignment tax' (loss of reasoning and creative diversity)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is human pairwise comparison significantly more reliable and cheaper to collect than expert demonstration writing?
  • ?
    What causes the 'alignment tax' during RLHF, and how does the KL divergence penalty mitigate it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-024: SFT Data Mixture and Multi-Task Balancing📋 Back to BankNext →M5-026: Choosing Learning Rates and Epochs in SFT