DPO vs GRPO Preference Alignment

Modern Preference Alignment & Reasoning RL algorithms—Direct Preference Optimization (DPO) vs Group Relative Policy Optimization (GRPO)—govern safety alignment and advanced logical reasoning in LLMs; 1) DPO: mathematically bypasses explicit Reward Model fitting and PPO loops by expressing optimal rewards directly via the policy ratio under the Bradley-Terry preference model, yielding a closed-form binary cross-entropy loss over $(x, y_w, y_l)$: $\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}[\ln \sigma(\beta \ln \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \ln \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)})]$; 2) GRPO (Group Relative Policy Optimization, pioneered by DeepSeek-Math/R1): discards the memory-heavy Critic value model entirely, sampling a group of $G$ outputs per prompt and standardizing relative advantages $A_i = \frac{r_i - \text{mean}(\{r\})}{\text{std}(\{r\})}$ within the cohort, slashing RL VRAM by 50% for long-chain reasoning tasks.

LLM helpfulness/safety alignment, long-chain mathematical reasoning bootstrapping (R1-style), and tone calibration.

Traditional PPO requires 4 concurrent models (Actor, Critic, Reward, Ref) in VRAM and suffers severe training instability; DPO and GRPO eliminate Critic bottlenecks to achieve stable, high-throughput preference optimization.

Derive the mathematical proof of DPO showing how the reward function is algebraically substituted with the log-policy ratio under Bradley-Terry?

Explain the role of hyperparameter $\beta$ in DPO in penalizing KL divergence drift away from the frozen reference policy?

Why does GRPO eliminate the Critic model, and how does it compute cohort-standardized advantages across $G$ sampled outputs?

How does rule-based reward verification (unit tests, exact math matching, format tags) induce long-chain reasoning emergence in DeepSeek-R1?

Explain the likelihood displacement pathology in DPO where both win/lose probabilities drop, and how KTO/IPO addresses it?