M5-043M5: NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)Easy
⚡Flashcard Drill➔Mastery:
Systematic Comparison of PPO, DPO, and GRPO
📐 Mathematical Definition
⚡ One-Sentence Core Answer
PPO uses an explicit Reward Model and Critic network for online policy gradients, DPO uses offline preference pairs to optimize an implicit reward without reinforcement learning, and GRPO eliminates the Critic by computing relative advantages across prompt rollout groups.
📌 Key Takeaways
- •PPO (Schulman et al. / OpenAI): 4 models (Actor, Critic, Reference, RM); online rollouts; highly flexible and gold standard for open-ended exploration, but complex and unstable
- •DPO (Rafailov et al. / Stanford): 2 models (Actor, Reference); 100% offline static preference pairs; stable supervised cross-entropy training, but limited by offline data distribution
- •GRPO (DeepSeekMath / DeepSeek-R1): 2 models (Actor, Reference); online rollouts with group sampling ( outputs per prompt); eliminates Critic by computing normalized intra-group advantage; dominant for math and code reasoning
📐 Mathematical Derivations
Mathematical Mechanism: Structural Comparison Matrix: 1. PPO: Requires Critic network to learn state-value baselines. 2. DPO: Zero online sampling; trains directly on offline dataset . 3. GRPO (Group Relative Policy Optimization): Samples completions per prompt . Advantage for sample is normalized directly against the group mean and standard deviation: No Critic model needed! Slashes training memory and eliminates Critic value divergence completely.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Infrastructure Memory Footprint: - PPO: Actor + Critic + Ref + RM base parameters in VRAM. - DPO: Actor + Ref base parameters (or if Ref is offloaded/quantized). - GRPO: Actor + Ref base parameters, but runs dynamic on-policy generation rollouts. ② Applicability Domains: - DPO is best for general alignment (conversational tone, safety, persona) where pairwise comparison data is readily available. - GRPO is best for verifiable reasoning (math, coding, logical puzzle solving) where rule-based execution environments provide scalar rewards and group exploration discovers novel reasoning paths. - PPO remains competitive in multi-objective complex environments where reward models are non-stationary and continuous RL exploration is mandatory. ③ Why GRPO Replaced PPO in DeepSeek-R1: Training a Critic model for long reasoning chains (10,000+ tokens) is virtually impossible because value predictions oscillate wildly at each intermediate step. GRPO bypasses the Critic entirely by using group baseline normalization. ⑤ Interview Strategy: Provide the 3-way taxonomy table across Models Required, Online vs Offline, Advantage Estimation Method, and Primary Use Cases, highlighting GRPO's role in modern reasoning models.
⚠️ Common Interview Pitfalls
- ✕Believing DPO is an online reinforcement learning algorithm (standard DPO is an offline supervised classification algorithm)
- ✕Attempting to use GRPO without group sampling (group size is mandatory to compute intra-group standard deviation)
- ✕Assuming PPO is obsolete (PPO remains superior in complex game-theoretic and multi-turn interactive environments)
🎯 Interviewer Follow-up Inquiries
- ?Why is training a Critic network especially intractable for 10,000-token Chain-of-Thought reasoning models?
- ?Under what conditions does offline DPO suffer compared to online GRPO exploration?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
Explore the comprehensive technical article, exam cards, and global architecture tree.
中文速览与面试要点
比较 PPO、DPO 与 GRPO 的依赖与适用场景。
PPO 需 RM+Critic+在线采样;DPO 只需偏好对与参考模型;GRPO 去 Critic、用组内基线、需可验证奖励。
核心要点
- 数学机理: 三者的依赖与机制 。 PPO(RLHF 标准) ——依赖:(a) 奖励模型 (RM,需偏好数据训练);(b) Critic (价值网络,需训练);(c) Reference (参考模型,算 KL);(d) 在线采样 (每轮用当前策略生成)。 优点 ——最强的能力(在线探索、可迭代改进、可处理任意奖励)。 缺点 ——工程复杂(4 模型、显存大)、训练不稳、采样慢、超参多。 DPO ——依赖:(a) 偏好对数据 ;(b) 参考模型 (通常是 SFT 模型)。 机制 ——直接把偏好对做对比损失(见推导),无需 RM、无需 RL 循环、无需采样。 优点 ——简单稳定(像 SFT)、成本低、易复现。 缺点 ——(a) 无在线探索 (用固定数据,无法探索数据外的区域);(b) 假设数据来自参考策略 (否则效果下降);(c) 无法利用'未通过验证'的样本(不像 RL 可用部分奖励)。 GRPO(Group Relative Policy Optimization) ——依赖:(a) 奖励函数 (可为 RM 或 可验证奖励 如答案对错);(b) 在线采样 (每个 prompt 采样一组回答); 去掉 Critic (用 组内平均奖励作为基线 )。 机制 ——对每个 prompt 采样 G 个回答,计算各自奖励,用 (r_i − mean)/std 作为优势(组内归一化),再用 PPO 式损失优化(但无 Critic)。 优点 ——比 PPO 省一个模型、比 DPO 有在线探索能力;特别适合 可验证奖励 (数学/代码:奖励是'答案是否正确')。 缺点 ——需采样(慢)、奖励需可靠、仍有 RL 的不稳定性。 选择逻辑 ——(a) 有偏好数据、求简单稳定 → DPO;(b) 有可验证奖励(数学/代码)、求推理能力 → GRPO/RLVR;(c) 需最强对齐、有充足资源 → PPO(+ 迭代 RM);(d) 实践路径 :DPO 起步 → 不足再上 GRPO/PPO。
- 深度剖析与工程权衡:① '在线探索'是 PPO/GRPO 相对 DPO 的核心优势 ——在线采样使策略能探索'偏好数据之外'的回答空间,从而发现更好的回答;DPO 只能在固定数据的'支撑集'内优化(这也是'DPO 难以超越数据质量'的原因)。② '奖励的来源'决定方法选择 ——(a) 有 人类偏好 → DPO/PPO;(b) 有 可验证奖励 (答案对错、测试通过) → GRPO/RLVR(最可靠);(c) 有 AI 反馈 → RLAIF 系列。故'先问奖励从哪来'是选择方法的第一步。③ GRPO 的'组内基线'为何有效 ——同一 prompt 的多个回答共享相同的'难度',故组内比较能消去'题目难度'这一混淆因素;这使基线估计更准(比 Critic 更简单且往往更好)。④ 与'推理模型'的关系 ——推理任务(数学/代码)天然有可验证奖励,故 GRPO/RLVR 成为推理模型训练的主流(DeepSeek-R1 等)。⑤ 成本对比 ——DPO(无采样、无 RM、无 Critic) 面试要点 ——被问'PPO/DPO/GRPO 怎么选',应给出' 依赖(RM/Critic/采样)+ 在线探索能力 + 奖励类型 + 成本 '的四维对比与选择逻辑;能指出'奖励来源决定方法选择'是深度理解的标志。
- PPO:最强但最复杂(4 模型、在线采样)
- DPO:最简单(无需 RM/RL),但无在线探索
- GRPO:介于两者(无 Critic,但需采样与奖励)
常见陷阱
- 认为 DPO 全面优于 PPO(DPO 无在线探索)
- 在可验证任务上用 DPO 而非 GRPO/RLVR