M5-043M5: NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)Easy
⚡Flashcard Drill➔Mastery:
Systematic Comparison of PPO, DPO, and GRPO
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
PPO uses an explicit Reward Model and Critic network for online policy gradients, DPO uses offline preference pairs to optimize an implicit reward without reinforcement learning, and GRPO eliminates the Critic by computing relative advantages across prompt rollout groups.
📌 Key Takeaways
- •PPO (Schulman et al. / OpenAI): 4 models (Actor, Critic, Reference, RM); online rollouts; highly flexible and gold standard for open-ended exploration, but complex and unstable
- •DPO (Rafailov et al. / Stanford): 2 models (Actor, Reference); 100% offline static preference pairs; stable supervised cross-entropy training, but limited by offline data distribution
- •GRPO (DeepSeekMath / DeepSeek-R1): 2 models (Actor, Reference); online rollouts with group sampling ( outputs per prompt); eliminates Critic by computing normalized intra-group advantage; dominant for math and code reasoning
📐 Mathematical Derivations
Mathematical Mechanism: Structural Comparison Matrix: 1. PPO: Requires Critic network to learn state-value baselines. 2. DPO: Zero online sampling; trains directly on offline dataset . 3. GRPO (Group Relative Policy Optimization): Samples completions per prompt . Advantage for sample is normalized directly against the group mean and standard deviation: No Critic model needed! Slashes training memory and eliminates Critic value divergence completely.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Infrastructure Memory Footprint: - PPO: Actor + Critic + Ref + RM base parameters in VRAM. - DPO: Actor + Ref base parameters (or if Ref is offloaded/quantized). - GRPO: Actor + Ref base parameters, but runs dynamic on-policy generation rollouts. ② Applicability Domains: - DPO is best for general alignment (conversational tone, safety, persona) where pairwise comparison data is readily available. - GRPO is best for verifiable reasoning (math, coding, logical puzzle solving) where rule-based execution environments provide scalar rewards and group exploration discovers novel reasoning paths. - PPO remains competitive in multi-objective complex environments where reward models are non-stationary and continuous RL exploration is mandatory. ③ Why GRPO Replaced PPO in DeepSeek-R1: Training a Critic model for long reasoning chains (10,000+ tokens) is virtually impossible because value predictions oscillate wildly at each intermediate step. GRPO bypasses the Critic entirely by using group baseline normalization. ⑤ Interview Strategy: Provide the 3-way taxonomy table across Models Required, Online vs Offline, Advantage Estimation Method, and Primary Use Cases, highlighting GRPO's role in modern reasoning models.
⚠️ Common Interview Pitfalls
- ✕Believing DPO is an online reinforcement learning algorithm (standard DPO is an offline supervised classification algorithm)
- ✕Attempting to use GRPO without group sampling (group size is mandatory to compute intra-group standard deviation)
- ✕Assuming PPO is obsolete (PPO remains superior in complex game-theoretic and multi-turn interactive environments)
🎯 Interviewer Follow-up Inquiries
- ?Why is training a Critic network especially intractable for 10,000-token Chain-of-Thought reasoning models?
- ?Under what conditions does offline DPO suffer compared to online GRPO exploration?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚