Direct Preference Optimization

DPO (Direct Preference Optimization) reparameterizes the KL-regularized RLHF objective into a closed-form loss over policy probability ratios, removing the reward model and the online RL loop. Derivation chain (3 steps) showing how the partition function $Z(x)$ cancels: ① The KL-regularized RLHF objective $\max_\pi \mathbb{E}_{x, y \sim \pi}[r(x,y)] - \beta\, D_{KL}(\pi \| \pi_{ref})$ has the closed-form optimum $\pi^*(y|x) = \frac{1}{Z(x)}\, \pi_{ref}(y|x)\, e^{r(x,y)/\beta}$, where the partition function $Z(x) = \sum_y \pi_{ref}(y|x)\, e^{r(x,y)/\beta}$ depends only on $x$, not on $y$; ② rearrange to get the implicit reward $r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$; ③ substitute into the Bradley-Terry preference model $P(y_w \succ y_l) = \sigma\left(r(x, y_w) - r(x, y_l)\right)$: subtracting the two terms cancels $\beta \log Z(x)$ exactly (since $Z$ does not depend on $y$), leaving a preference probability over ratios — $$P(y_w \succ y_l) = \sigma\left( \beta \log \frac{\pi^*(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{ref}(y_l|x)} \right)$$ Replacing $\pi^*$ with $\pi_\theta$ and maximizing the likelihood on preference data gives the DPO loss: $$\mathcal{L}_{DPO}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$ $y_w$/$y_l$ are the chosen/rejected of a preference pair; $\pi_{ref}$ is the reference policy (usually the SFT model); $\beta$ is the temperature/regularization strength: larger $\beta$ keeps the policy closer to $\pi_{ref}$ with smaller KL but weaker preference exploitation; smaller $\beta$ fits preferences aggressively with higher gains but risks overfitting, style drift, and degradation (e.g. length bias and reward mimicry).

offline preference alignment (safety, helpfulness, style) whenever contrastive preference pairs exist; the 'SFT → DPO' recipe is the low-cost open-source default, and DPO can be mixed with GRPO (online RLVR + offline preference regularization).

RLHF's four-stage pipeline (reward-model training → online sampling → PPO → KL tuning) is complex, unstable, and memory-hungry; DPO performs alignment in one supervised step — no reward model, no RL loop, no online sampling, with dramatically lower memory and compute — at the cost that preference-data quality directly determines the outcome: label noise, length bias, and reference-model bias are amplified.

Write the DPO loss and explain every symbol (y_w/y_l/π_ref/β)?

Derivation: how does Z(x) cancel? Give the closed-form solution and the 3-step chain?

What does β mean? Effects of large vs small β?

Why does DPO need no reward model or RL? Its relation to the RLHF objective?

DPO vs GRPO selection criteria? DPO's main limitations (data quality, length bias, role of the reference model)?