M5-043M5: NLP & Large Language ModelsDPO Family (DPO / KTO / ORPO)Easy
⚡Flashcard Drill➔Mastery:

Systematic Comparison of PPO, DPO, and GRPO

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-044 →
📐 Mathematical Definition
PPO: RM+Critic+on-policy;DPO: pairs+πref;GRPO: group baseline, no Critic\text{PPO}:\ \text{RM}+\text{Critic}+\text{on-policy};\quad \text{DPO}:\ \text{pairs}+\pi_{\text{ref}};\quad \text{GRPO}:\ \text{group baseline},\ \text{no Critic}
⚡ One-Sentence Core Answer
PPO uses an explicit Reward Model and Critic network for online policy gradients, DPO uses offline preference pairs to optimize an implicit reward without reinforcement learning, and GRPO eliminates the Critic by computing relative advantages across prompt rollout groups.

📌 Key Takeaways

  • •
    PPO (Schulman et al. / OpenAI): 4 models (Actor, Critic, Reference, RM); online rollouts; highly flexible and gold standard for open-ended exploration, but complex and unstable
  • •
    DPO (Rafailov et al. / Stanford): 2 models (Actor, Reference); 100% offline static preference pairs; stable supervised cross-entropy training, but limited by offline data distribution
  • •
    GRPO (DeepSeekMath / DeepSeek-R1): 2 models (Actor, Reference); online rollouts with group sampling (GG outputs per prompt); eliminates Critic by computing normalized intra-group advantage; dominant for math and code reasoning

📐 Mathematical Derivations

Mathematical Mechanism: Structural Comparison Matrix: 1. PPO: LPPO=E^t[min⁡(rtA^tGAE,clip(rt,1−ϵ,1+ϵ)A^tGAE)],A^t=Q(st,at)−Vψ(st)\mathcal{L}_{\text{PPO}} = \hat{\mathbb{E}}_t \left[ \min(r_t \hat{A}_t^{\text{GAE}}, \text{clip}(r_t, 1-\epsilon, 1+\epsilon) \hat{A}_t^{\text{GAE}}) \right], \quad \hat{A}_t = Q(s_t, a_t) - V_\psi(s_t) Requires Critic network VψV_\psi to learn state-value baselines. 2. DPO: LDPO=−E(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right] Zero online sampling; trains directly on offline dataset D\mathcal{D}. 3. GRPO (Group Relative Policy Optimization): Samples GG completions {y1,…,yG}∼πθold(⋅∣x)\{y_1, \dots, y_G\} \sim \pi_{\theta_{\text{old}}}(\cdot \mid x) per prompt xx. Advantage for sample ii is normalized directly against the group mean and standard deviation: A^i=Ri−mean({R1,…,RG})std({R1,…,RG})+ϵ\hat{A}_i = \frac{R_i - \text{mean}(\{R_1, \dots, R_G\})}{\text{std}(\{R_1, \dots, R_G\}) + \epsilon} No Critic model needed! Slashes training memory and eliminates Critic value divergence completely.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① Infrastructure Memory Footprint: - PPO: Actor + Critic + Ref + RM   ⟹  ∼4×\implies \sim 4\times base parameters in VRAM. - DPO: Actor + Ref   ⟹  ∼2×\implies \sim 2\times base parameters (or 1×1\times if Ref is offloaded/quantized). - GRPO: Actor + Ref   ⟹  ∼2×\implies \sim 2\times base parameters, but runs dynamic on-policy generation rollouts. ② Applicability Domains: - DPO is best for general alignment (conversational tone, safety, persona) where pairwise comparison data is readily available. - GRPO is best for verifiable reasoning (math, coding, logical puzzle solving) where rule-based execution environments provide scalar rewards and group exploration discovers novel reasoning paths. - PPO remains competitive in multi-objective complex environments where reward models are non-stationary and continuous RL exploration is mandatory. ③ Why GRPO Replaced PPO in DeepSeek-R1: Training a Critic model for long reasoning chains (10,000+ tokens) is virtually impossible because value predictions oscillate wildly at each intermediate step. GRPO bypasses the Critic entirely by using group baseline normalization. ⑤ Interview Strategy: Provide the 3-way taxonomy table across Models Required, Online vs Offline, Advantage Estimation Method, and Primary Use Cases, highlighting GRPO's role in modern reasoning models.
⚠️ Common Interview Pitfalls
  • ✕
    Believing DPO is an online reinforcement learning algorithm (standard DPO is an offline supervised classification algorithm)
  • ✕
    Attempting to use GRPO without group sampling (group size G≥4G \ge 4 is mandatory to compute intra-group standard deviation)
  • ✕
    Assuming PPO is obsolete (PPO remains superior in complex game-theoretic and multi-turn interactive environments)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is training a Critic network especially intractable for 10,000-token Chain-of-Thought reasoning models?
  • ?
    Under what conditions does offline DPO suffer compared to online GRPO exploration?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-042: Full Mathematical Derivation of Direct Preference Optimization (DPO) Loss📋 Back to BankNext →M5-044: Motivations and Distinctions of ORPO and KTO