M5-030M5: NLP & Large Language ModelsAlignment & RLHFEasy
⚡Flashcard Drill➔Mastery:
The Three-Stage RLHF Pipeline
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The canonical RLHF pipeline consists of three sequential phases: Supervised Fine-Tuning to establish instruction following, Reward Modeling on human preference pairs, and PPO Reinforcement Learning with a KL penalty to optimize policy generation.
📌 Key Takeaways
- •Stage 1: Supervised Fine-Tuning (SFT) establishes a competent initial policy capable of following instructions and generating well-formatted responses
- •Stage 2: Reward Modeling (RM) trains a scoring network on human pairwise preferences using the Bradley-Terry ranking loss
- •Stage 3: Reinforcement Learning (PPO) optimizes policy to maximize reward while constraining policy divergence via a per-token KL penalty against
📐 Mathematical Derivations
Mathematical Mechanism: 1. Stage 1: SFT Warmup: Standard cross-entropy optimization over demonstration dataset : 2. Stage 2: Bradley-Terry Reward Modeling: Given prompt and human-labeled preference , train scalar reward model via negative log-likelihood: 3. Stage 3: PPO Policy Optimization: Optimize policy to maximize expected reward subject to KL divergence regularization: Where per-token KL divergence acts as an immediate penalty: .
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Why Comparative Preference over Absolute Scoring: Absolute ratings (1 to 7 scale) suffer from severe inter-annotator bias (one annotator's '5' is another's '7'). Pairwise comparison () is intuitive, consistent, and achieves significantly higher inter-annotator agreement. ② The Role of the KL Penalty $\beta$: Without the KL divergence penalty (), the policy quickly exploits reward model blind spots, generating ungrammatical, verbose gibberish that scores high reward (reward hacking). keeps the policy anchored within the natural language distribution of . ③ Infrastructure Complexity of PPO: Stage 3 requires hosting four large models in GPU memory simultaneously: Actor (policy ), Critic (value function ), Reference Model (frozen ), and Reward Model (), requiring complex multi-GPU distributed orchestration (DeepSpeed-Chat, Ray, vLLM). ④ Direct Preference Optimization (DPO) as an Alternative: DPO mathematically reparameterizes the reward model directly into the policy loss, eliminating the need to train a separate reward model or run PPO loops. ⑤ Interview Strategy: Diagram the three stages chronologically, write down the Bradley-Terry and PPO-KL optimization formulas, and explain the essential stabilizing role of the KL constraint.
⚠️ Common Interview Pitfalls
- ✕Attempting PPO optimization without a KL constraint (causes immediate reward hacking and language degradation)
- ✕Skipping Stage 1 SFT and attempting to run RL on a raw base model
- ✕Assuming reward model absolute scalar scores have meaningful units (only the relative difference is identifiable)
🎯 Interviewer Follow-up Inquiries
- ?Why is human pairwise comparison significantly more consistent than absolute scalar Likert-scale scoring?
- ?How does DPO mathematically eliminate the need for an explicit Reward Model and PPO training loop?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚