M5-030M5: NLP & Large Language ModelsAlignment & RLHFEasy
⚡Flashcard Drill➔Mastery:

The Three-Stage RLHF Pipeline

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-031 →
📐 Mathematical Definition
SFT→RM: L=−log⁡σ(rϕ(x,yw)−rϕ(x,yl))→PPO: max⁡E[rϕ]−βKL(π∥πref)\text{SFT}\to\text{RM}:\ \mathcal{L}=-\log\sigma(r_\phi(x,y_w)-r_\phi(x,y_l))\to\text{PPO}:\ \max\mathbb{E}[r_\phi]-\beta\mathrm{KL}(\pi\|\pi_{\text{ref}})
⚡ One-Sentence Core Answer
The canonical RLHF pipeline consists of three sequential phases: Supervised Fine-Tuning to establish instruction following, Reward Modeling on human preference pairs, and PPO Reinforcement Learning with a KL penalty to optimize policy generation.

📌 Key Takeaways

  • •
    Stage 1: Supervised Fine-Tuning (SFT) establishes a competent initial policy πSFT\pi_{\text{SFT}} capable of following instructions and generating well-formatted responses
  • •
    Stage 2: Reward Modeling (RM) trains a scoring network rϕ(x,y)r_\phi(x, y) on human pairwise preferences (yw≻yl)(y_w \succ y_l) using the Bradley-Terry ranking loss
  • •
    Stage 3: Reinforcement Learning (PPO) optimizes policy πθ\pi_\theta to maximize reward rϕ(x,y)r_\phi(x, y) while constraining policy divergence via a per-token KL penalty against πSFT\pi_{\text{SFT}}

📐 Mathematical Derivations

Mathematical Mechanism: 1. Stage 1: SFT Warmup: Standard cross-entropy optimization over demonstration dataset DSFT\mathcal{D}_{\text{SFT}}: πSFT=arg⁡max⁡θE(x,y)∼DSFT[∑tlog⁡πθ(yt∣x,y<t)]\pi_{\text{SFT}} = \arg\max_\theta \mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{SFT}}} \left[ \sum_{t} \log \pi_\theta(y_t \mid x, y_{<t}) \right] 2. Stage 2: Bradley-Terry Reward Modeling: Given prompt xx and human-labeled preference yw≻yly_w \succ y_l, train scalar reward model rϕ(x,y)r_\phi(x, y) via negative log-likelihood: LRM(ϕ)=−E(x,yw,yl)[log⁡σ(rϕ(x,yw)−rϕ(x,yl))]\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( r_\phi(x, y_w) - r_\phi(x, y_l) \right) \right] 3. Stage 3: PPO Policy Optimization: Optimize policy πθ\pi_\theta to maximize expected reward subject to KL divergence regularization: max⁡θEx∼D,y∼πθ(⋅∣x)[rϕ(x,y)−βDKL(πθ(y∣x) ∥ πSFT(y∣x))]\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot \mid x)} \left[ r_\phi(x, y) - \beta D_{\text{KL}}\left(\pi_\theta(y \mid x) \, \| \, \pi_{\text{SFT}}(y \mid x)\right) \right] Where per-token KL divergence acts as an immediate penalty: rtotal(x,y)=rϕ(x,y)−β∑tlog⁡πθ(yt∣x,y<t)πSFT(yt∣x,y<t)r_{\text{total}}(x, y) = r_\phi(x, y) - \beta \sum_{t} \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\text{SFT}}(y_t \mid x, y_{<t})}.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Why Comparative Preference over Absolute Scoring: Absolute ratings (1 to 7 scale) suffer from severe inter-annotator bias (one annotator's '5' is another's '7'). Pairwise comparison (yw≻yly_w \succ y_l) is intuitive, consistent, and achieves significantly higher inter-annotator agreement. ② The Role of the KL Penalty $\beta$: Without the KL divergence penalty (β=0\beta = 0), the policy quickly exploits reward model blind spots, generating ungrammatical, verbose gibberish that scores high reward (reward hacking). β\beta keeps the policy anchored within the natural language distribution of πSFT\pi_{\text{SFT}}. ③ Infrastructure Complexity of PPO: Stage 3 requires hosting four large models in GPU memory simultaneously: Actor (policy πθ\pi_\theta), Critic (value function VψV_\psi), Reference Model (frozen πSFT\pi_{\text{SFT}}), and Reward Model (rϕr_\phi), requiring complex multi-GPU distributed orchestration (DeepSpeed-Chat, Ray, vLLM). ④ Direct Preference Optimization (DPO) as an Alternative: DPO mathematically reparameterizes the reward model directly into the policy loss, eliminating the need to train a separate reward model or run PPO loops. ⑤ Interview Strategy: Diagram the three stages chronologically, write down the Bradley-Terry and PPO-KL optimization formulas, and explain the essential stabilizing role of the KL constraint.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting PPO optimization without a KL constraint (causes immediate reward hacking and language degradation)
  • ✕
    Skipping Stage 1 SFT and attempting to run RL on a raw base model
  • ✕
    Assuming reward model absolute scalar scores have meaningful units (only the relative difference r(yw)−r(yl)r(y_w) - r(y_l) is identifiable)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is human pairwise comparison significantly more consistent than absolute scalar Likert-scale scoring?
  • ?
    How does DPO mathematically eliminate the need for an explicit Reward Model and PPO training loop?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-029: SFT Data Sources and Engineering Trade-offs📋 Back to BankNext →M5-031: Bradley-Terry Preference Model and Reward Model Loss Formulation