Direct Preference Optimization

DPO (Direct Preference Optimization) reparameterizes the KL-regularized RLHF objective into a closed-form loss over policy probability ratios, removing the reward model and the online RL loop. Derivation chain (3 steps) showing how the partition function $Z(x)$ cancels: ① The KL-regularized RLHF objective $\max_\pi \mathbb{E}_{x, y \sim \pi}[r(x,y)] - \beta\, D_{KL}(\pi \| \pi_{ref})$ has the closed-form optimum $\pi^*(y|x) = \frac{1}{Z(x)}\, \pi_{ref}(y|x)\, e^{r(x,y)/\beta}$, where the partition function $Z(x) = \sum_y \pi_{ref}(y|x)\, e^{r(x,y)/\beta}$ depends only on $x$, not on $y$; ② rearrange to get the implicit reward $r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$; ③ substitute into the Bradley-Terry preference model $P(y_w \succ y_l) = \sigma\left(r(x, y_w) - r(x, y_l)\right)$: subtracting the two terms cancels $\beta \log Z(x)$ exactly (since $Z$ does not depend on $y$), leaving a preference probability over ratios — $$P(y_w \succ y_l) = \sigma\left( \beta \log \frac{\pi^*(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{ref}(y_l|x)} \right)$$ Replacing $\pi^*$ with $\pi_\theta$ and maximizing the likelihood on preference data gives the DPO loss: $$\mathcal{L}_{DPO}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$ $y_w$/$y_l$ are the chosen/rejected of a preference pair; $\pi_{ref}$ is the reference policy (usually the SFT model); $\beta$ is the temperature/regularization strength: larger $\beta$ keeps the policy closer to $\pi_{ref}$ with smaller KL but weaker preference exploitation; smaller $\beta$ fits preferences aggressively with higher gains but risks overfitting, style drift, and degradation (e.g. length bias and reward mimicry).

offline preference alignment (safety, helpfulness, style) whenever contrastive preference pairs exist; the 'SFT → DPO' recipe is the low-cost open-source default, and DPO can be mixed with GRPO (online RLVR + offline preference regularization).

RLHF's four-stage pipeline (reward-model training → online sampling → PPO → KL tuning) is complex, unstable, and memory-hungry; DPO performs alignment in one supervised step — no reward model, no RL loop, no online sampling, with dramatically lower memory and compute — at the cost that preference-data quality directly determines the outcome: label noise, length bias, and reference-model bias are amplified.

Test Your Knowledge: Practice Questions for "Direct Preference Optimization"

Single choice pitfall questions with instant feedback and mistake tracking.