M5-025M5: NLP & Large Language ModelsInstruction Tuning & Supervised Fine-TuningHard
⚡Flashcard Drill➔Mastery:
Division of Labor Between SFT and RLHF/DPO
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
SFT establishes foundational instruction following, domain knowledge, and response formatting, whereas RLHF/DPO refines preference boundaries, suppresses low-probability tail errors, and aligns model outputs with human values.
📌 Key Takeaways
- •SFT role: builds the baseline capability to understand user prompts and respond in coherent, structured markdown; unlocks latent pre-training capabilities
- •RLHF / DPO role: explores the solution space, optimizes relative preference nuances, suppresses hallucinations and toxic completions, and resolves edge-case ambiguity
- •Complementary duality: SFT sets the high-quality starting policy ; RL cannot function without a competent SFT base to explore from
📐 Mathematical Derivations
Mathematical Mechanism: 1. SFT as Maximum Likelihood Estimation (MLE): SFT maximizes data likelihood over positive demonstrations: MLE treats all demonstration tokens equally. It has no negative feedback mechanism: it cannot explicitly penalize toxic, hallucinated, or sub-optimal answers. 2. RLHF / DPO as Contrastive Optimization: Given prompt and pair (winning vs losing response): RLHF/DPO simultaneously pushes up the likelihood of preferred completions while actively pushing down the likelihood of dispreferred completions, carving sharp decision boundaries.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Why RL Cannot Replace SFT: If you initialize RLHF or DPO directly from a raw pre-trained base model, the action space (all vocabulary tokens across length ) is astronomically vast. The policy generates unformatted, rambling text that receives near-zero reward, resulting in random gradient noise and optimization failure. SFT provides the vital warm-start policy . ② Why SFT Cannot Replace RL: In complex tasks, human evaluators can easily pick the better of two answers (), but struggle to write a perfect reference answer from scratch. RLHF leverages pairwise comparison data to surpass the demonstration ability of average human annotators. ③ Mode Collapse vs Exploration: SFT suffers from exposure bias (teacher forcing). RL allows the model to sample its own generations autoregressively and receive feedback, learning error correction. ④ The Alignment Tax: Excessive RLHF/DPO can degrade creative writing diversity and multi-step reasoning (the alignment tax), requiring careful KL regularization against . ⑤ Interview Strategy: Contrast MLE (positive-only) vs Contrastive RL (positive and negative), explain why SFT is the prerequisite warm-start for RL, and articulate the complementary strengths of both phases.
⚠️ Common Interview Pitfalls
- ✕Attempting to run RLHF or DPO directly on a raw base model without SFT warm-start
- ✕Assuming SFT can learn negative constraints effectively (cross-entropy cannot explicitly penalize dispreferred answers without contrastive objectives)
- ✕Pushing RL optimization so aggressively that the model suffers the 'alignment tax' (loss of reasoning and creative diversity)
🎯 Interviewer Follow-up Inquiries
- ?Why is human pairwise comparison significantly more reliable and cheaper to collect than expert demonstration writing?
- ?What causes the 'alignment tax' during RLHF, and how does the KL divergence penalty mitigate it?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚