F2Hard★ Core Essential · Industrial BedrockPart F · Alignment & Reinforcement Learning
Direct Preference Optimization (DPO)
Optimize policy alignment directly on preference pairs via implicit rewards, bypassing RL rollouts and reward models.
⏱️ Time:
O(B * S) 前向计算对数似然💾 Space:
需在显存中保留 Reference 模型的权重(通常采用 LoRA 共享基底以节省显存)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def dpo_loss(
policy_chosen_logps: np.ndarray, # (B,) 当前策略在 y_w 上的序列 log 概率之和
policy_rejected_logps: np.ndarray, # (B,) 当前策略在 y_l 上的序列 log 概率之和
reference_chosen_logps: np.ndarray,# (B,) 冻结参考策略在 y_w 上的 log 概率之和
reference_rejected_logps: np.ndarray,
beta: float = 0.1
) -> float:
"""
参数:
beta: KL 惩罚强度系数 (常用 0.05 ~ 0.2)
"""
# 1. 计算隐式奖励比率
chosen_ratio =
rejected_ratio = policy_rejected_logps - reference_rejected_logps
# 2. 构造隐式奖励差 logits
logits = beta * (chosen_ratio - rejected_ratio)
# 3. 稳定计算 -log(sigmoid(logits)) = logaddexp(0, -logits)
losses = np.logaddexp(0.0, -logits)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does the absolute log-probability of Chosen responses sometimes decrease during DPO training?
💡
Core Mental Anchor / Mnemonic
Compare policy-to-ref log-ratios of chosen vs. rejected completions inside logaddexp.
📐 Mathematical Derivation & Core Formula
### Implicit Reward Reparameterization
Rafailov et al. (2023) showed the KL-constrained RLHF objective has the closed-form optimal reward relationship:
Substituting into the Bradley-Terry likelihood cancels the intractable partition function :
This eliminates explicit reward model training and online RL rollouts, optimizing policy alignment directly from offline pairs.
Rafailov et al. (2023) showed the KL-constrained RLHF objective has the closed-form optimal reward relationship:
Substituting into the Bradley-Terry likelihood cancels the intractable partition function :
This eliminates explicit reward model training and online RL rollouts, optimizing policy alignment directly from offline pairs.
🔄 Tensor Dimensions & Shape Flow
policy & ref logprobs -> log-ratio margins -> beta * (chosen_margin - rejected_margin) -> logaddexp -> scalar
🛡️ Numerical Stability & Pitfalls Checklist
- Sum token log-probabilities strictly over completion tokens, masking out prompt tokens.
- Very small beta weakens preference separation; overly large beta pins policy too tightly to reference.
- Monitor chosen vs. rejected log-prob trajectories for absolute likelihood decline (addressed in SimPO/ORPO).
💻 Reference Code Implementation
import numpy as np
def dpo_loss(
policy_chosen_logps: np.ndarray, # (B,) 当前策略在 y_w 上的序列 log 概率之和
policy_rejected_logps: np.ndarray, # (B,) 当前策略在 y_l 上的序列 log 概率之和
reference_chosen_logps: np.ndarray,# (B,) 冻结参考策略在 y_w 上的 log 概率之和
reference_rejected_logps: np.ndarray,
beta: float = 0.1
) -> float:
"""
参数:
beta: KL 惩罚强度系数 (常用 0.05 ~ 0.2)
"""
# 1. 计算隐式奖励比率
chosen_ratio = policy_chosen_logps - reference_chosen_logps
rejected_ratio = policy_rejected_logps - reference_rejected_logps
# 2. 构造隐式奖励差 logits
logits = beta * (chosen_ratio - rejected_ratio)
# 3. 稳定计算 -log(sigmoid(logits)) = logaddexp(0, -logits)
losses = np.logaddexp(0.0, -logits)
return float(np.mean(losses))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
# 模拟:策略对 chosen 赋予更高概率,对 rejected 赋予更低概率
pi_w = np.array([-10.0])
pi_l = np.array([-25.0])
ref_w = np.array([-15.0])
ref_l = np.array([-20.0])
loss = dpo_loss(pi_w, pi_l, ref_w, ref_l, beta=0.1)
# chosen 提升 5,rejected 下降 5,总差值 +10,logits = 0.1 * 10 = 1.0
# loss = ln(1 + e^-1) = 0.313
assert np.isclose(loss, 0.31326, atol=1e-3)
print("✓ DPO 损失函数自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why does the absolute log-probability of Chosen responses sometimes decrease during DPO training?
DPO constrains only the relative log-ratio margin between Chosen and Rejected completions. Decreasing Rejected probabilities often yields easier gradient reduction than lifting Chosen probabilities, causing both likelihoods to drift downward. Methods like SimPO or adding an SFT/NLL term mitigate this.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.