F1Medium★ Core Essential · Industrial BedrockPart F · Alignment & Reinforcement Learning
Bradley-Terry Preference Reward Model Loss
Pairwise logistic preference loss training reward models to score chosen responses above rejected ones.
⏱️ Time:
O(B)💾 Space:
O(B)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def bradley_terry_reward_loss(
r_chosen: np.ndarray, # (B,) 模型对胜出回复预测的标量奖励打分
r_rejected: np.ndarray # (B,) 模型对落败回复预测的标量奖励打分
) -> float:
"""
数值稳定的 Bradley-Terry 奖励模型损失计算。
"""
diff =
# -log(sigmoid(diff)) = log(1 + exp(-diff)) = logaddexp(0, -diff)
losses = np.logaddexp(0.0, -diff)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How can Bradley-Terry loss be extended when human preference labels include ties?
💡
Core Mental Anchor / Mnemonic
Feed -(r_chosen - r_rejected) into logaddexp(0, .) for numerically stable pairwise likelihood.
📐 Mathematical Derivation & Core Formula
### Pairwise Preference Likelihood Derivation
Bradley-Terry (1952) models the probability that chosen response beats rejected response under prompt via the Sigmoid of their scalar reward difference:
Maximizing log-likelihood minimizes:
Bradley-Terry (1952) models the probability that chosen response beats rejected response under prompt via the Sigmoid of their scalar reward difference:
Maximizing log-likelihood minimizes:
🔄 Tensor Dimensions & Shape Flow
r_chosen, r_rejected: (B,) -> diff: (B,) -> np.logaddexp(0, -diff) -> mean scalar
🛡️ Numerical Stability & Pitfalls Checklist
- Compute
-log(sigmoid(r_w - r_l))aslogaddexp(0, -(r_w - r_l))to avoid underflow when r_w << r_l. - Production RLHF often adds tiny score regularization (e.g., 1e-3 * (r_w^2 + r_l^2)) to prevent reward drift.
💻 Reference Code Implementation
import numpy as np
def bradley_terry_reward_loss(
r_chosen: np.ndarray, # (B,) 模型对胜出回复预测的标量奖励打分
r_rejected: np.ndarray # (B,) 模型对落败回复预测的标量奖励打分
) -> float:
"""
数值稳定的 Bradley-Terry 奖励模型损失计算。
"""
diff = r_chosen - r_rejected # (B,)
# -log(sigmoid(diff)) = log(1 + exp(-diff)) = logaddexp(0, -diff)
losses = np.logaddexp(0.0, -diff)
return float(np.mean(losses))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
r_w = np.array([3.0, 5.0])
r_l = np.array([1.0, 1.0])
loss = bradley_terry_reward_loss(r_w, r_l)
# 差值全为正,loss 应显著小于 ln(2) 约 0.693
assert loss < 0.2
print("✓ Bradley-Terry 奖励模型损失自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How can Bradley-Terry loss be extended when human preference labels include ties?
Use the Rao-Kupper extension with a tie-margin parameter , modeling tie probability explicitly when falls within the indifference region, or split ties into two half-weighted opposing pairs.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.