F1Medium★ Core Essential · Industrial BedrockPart F · Alignment & Reinforcement Learning

Bradley-Terry Preference Reward Model Loss

⚡Flashcard Drill➔

Pairwise logistic preference loss training reward models to score chosen responses above rejected ones.

⏱️ Time: O(B)
💾 Space: O(B)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def bradley_terry_reward_loss(
r_chosen: np.ndarray, # (B,) 模型对胜出回复预测的标量奖励打分
r_rejected: np.ndarray # (B,) 模型对落败回复预测的标量奖励打分
) -> float:
"""
数值稳定的 Bradley-Terry 奖励模型损失计算。
"""
diff =
# -log(sigmoid(diff)) = log(1 + exp(-diff)) = logaddexp(0, -diff)
losses = np.logaddexp(0.0, -diff)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How can Bradley-Terry loss be extended when human preference labels include ties?
Next: Direct Preference Optimization (DPO) →
💡

Core Mental Anchor / Mnemonic

Feed -(r_chosen - r_rejected) into logaddexp(0, .) for numerically stable pairwise likelihood.

📐 Mathematical Derivation & Core Formula

LRM=−E(x,yw,yl)[log⁡σ(rθ(x,yw)−rθ(x,yl))]\mathcal{L}_{\mathrm{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]
### Pairwise Preference Likelihood Derivation
Bradley-Terry (1952) models the probability that chosen response ywy_w beats rejected response yly_l under prompt xx via the Sigmoid of their scalar reward difference:
P(yw≻yl∣x)=σ(r(x,yw)−r(x,yl))=11+e−(rw−rl)P(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l)) = \frac{1}{1 + e^{-(r_w - r_l)}}
Maximizing log-likelihood minimizes:
L=−log⁡σ(rw−rl)=logaddexp(0,−(rw−rl))\mathcal{L} = -\log \sigma(r_w - r_l) = \mathrm{logaddexp}(0, -(r_w - r_l))

🔄 Tensor Dimensions & Shape Flow

r_chosen, r_rejected: (B,) -> diff: (B,) -> np.logaddexp(0, -diff) -> mean scalar

🛡️ Numerical Stability & Pitfalls Checklist

  • Compute -log(sigmoid(r_w - r_l)) as logaddexp(0, -(r_w - r_l)) to avoid underflow when r_w << r_l.
  • Production RLHF often adds tiny score regularization (e.g., 1e-3 * (r_w^2 + r_l^2)) to prevent reward drift.

💻 Reference Code Implementation

import numpy as np

def bradley_terry_reward_loss(
    r_chosen: np.ndarray,    # (B,) 模型对胜出回复预测的标量奖励打分
    r_rejected: np.ndarray   # (B,) 模型对落败回复预测的标量奖励打分
) -> float:
    """
    数值稳定的 Bradley-Terry 奖励模型损失计算。
    """
    diff = r_chosen - r_rejected  # (B,)
    # -log(sigmoid(diff)) = log(1 + exp(-diff)) = logaddexp(0, -diff)
    losses = np.logaddexp(0.0, -diff)
    return float(np.mean(losses))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
r_w = np.array([3.0, 5.0])
r_l = np.array([1.0, 1.0])
loss = bradley_terry_reward_loss(r_w, r_l)
# 差值全为正,loss 应显著小于 ln(2) 约 0.693
assert loss < 0.2
print("✓ Bradley-Terry 奖励模型损失自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How can Bradley-Terry loss be extended when human preference labels include ties?
Use the Rao-Kupper extension with a tie-margin parameter τ\tau, modeling tie probability explicitly when ∣rw−rl∣|r_w - r_l| falls within the indifference region, or split ties into two half-weighted opposing pairs.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Huber Loss & Smooth L1 LossBack to Part F & All 105Next: Direct Preference Optimization (DPO) →