M1Medium★ 核心必练 · 评测校准Part M · LLM 评测与 Agent/RAG 算子

LLM-as-a-Judge 位置消偏与 Cohen's Kappa 校准

⚡闪卡盲测➔

通过正反位置交换检测成对裁判的位置偏置,并用 Cohen's Kappa 扣除随机一致率度量人机对齐度。

⏱️ 时间复杂度: O(N * |Labels|)
💾 空间复杂度: O(N)
🎯 免登录实战自测闭环(先做再看答案)
按「机制回忆 → 关键行填空 → 边界反例 → 追问与错题记录」顺序自测,进度自动保存在本地。
Step 1 · 机制口述与数值稳定性自检
Step 2 · 核心代码关键行补全
import numpy as np
def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
"""
Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
"""
if vote_ab == "A" and vote_ba == "B":
return "A"
if vote_ab == "B" and vote_ba == "A":
return "B"
return "TIE"
def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
"""
Compute Cohen's Kappa agreement between LLM judge and human annotators.
"""
r1 = np.asarray(rater1)
r2 = np.asarray(rater2)
n = len(r1)
if n == 0:
return 0.0
if labels is None:
labels = sorted(set(r1.tolist()) | set(r2.tolist()))
p_o =
p_e = 0.0
for lbl in labels:
p1 = float(np.mean(r1 == lbl))
p2 = float(np.mean(r2 == lbl))
p_e += p1 * p2
if abs(1.0 - p_e) < 1e-12:
return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
return
Step 3 · 反例与边界断言核对
Step 4 · 面试追问盲答与错题记录
Q1: 除了 Position Bias,LLM-as-a-Judge 还有哪两类常见系统性偏差?工程上怎么缓解?
Q2: 成对比较(Pairwise)与单条打分(Pointwise 1–5 分)在回归门控里各自适合什么场景?
下一题: FSM 状态机结构化约束解码(Logit Masking) →
💡

实战一句话记忆口诀

正反互换查首位偏置,卡帕扣掉随机巧合,分母近零防除零

📐 核心数学公式与机理推导

κ=po−pe1−pe,pe=∑knk,1⋅nk,2N2\kappa = \frac{p_o - p_e}{1 - p_e}, \quad p_e = \sum_{k} \frac{n_{k,1} \cdot n_{k,2}}{N^2}
### 位置偏置消偏与机遇校正一致率
大模型裁判在成对比较 (A,B)(A, B) 时常偏好第一个出现的回答(Position Bias)。将候选顺序互换后再评一次 (B,A)(B, A):
- 若正序选首位 (A)、反序仍选首位 (A 位置对应原 B),说明存在位置偏置,判定为 TIE 或触发人工/更强模型仲裁;
- 只有两次结果指向同一实质候选时才计为有效胜出。

为什么不能只看原始准确率(Raw Agreement $p_o$)?
若类别极不平衡(如 90% 都是胜出或合格),盲猜多数类也能拿到 82% 以上的准确率。Cohen's Kappa 扣除双方独立随机一致的概率 pe=∑kP1(k)P2(k)p_e = \sum_k P_1(k) P_2(k):
κ=po−pe1−pe\kappa = \frac{p_o - p_e}{1 - p_e}
只有当 κ>0.6\kappa > 0.6(常见门槛 ≥0.7\ge 0.7)时,自动裁判才适合替代人工跑回归门控。

🔄 张量维度与形状流转说明

(vote_ab, vote_ba) -> 顺序翻转一致性检查 -> verdict; (human, judge) -> 边际概率求和得 p_e -> (p_o - p_e)/(1 - p_e) -> kappa

🛡️ 数值稳定性自检与边界要点

  • 位置互换后若两次都判首位胜出(A, A),决不能取其中一次结果,必须降级为平局或冲突样本
  • 当某一方全部输出同一标签且 pe→1p_e \to 1 时,分母 1−pe1 - p_e 接近 0,必须做零分母保护
  • 评测提示词与少样本示例顺序也应平衡正负例,避免引入提示词锚定偏差

💻 完整参考代码实现

import numpy as np

def resolve_position_swap(vote_ab: str, vote_ba: str) -> str:
    """
    Resolve pairwise LLM-as-a-Judge verdicts after swapping candidate order (A, B) -> (B, A).
    Votes are 'A', 'B', or 'TIE'. Only consistent winners survive; flipped votes become 'TIE'.
    """
    if vote_ab == "A" and vote_ba == "B":
        return "A"
    if vote_ab == "B" and vote_ba == "A":
        return "B"
    return "TIE"

def cohens_kappa(rater1: list, rater2: list, labels: list = None) -> float:
    """
    Compute Cohen's Kappa agreement between LLM judge and human annotators.
    """
    r1 = np.asarray(rater1)
    r2 = np.asarray(rater2)
    n = len(r1)
    if n == 0:
        return 0.0
    if labels is None:
        labels = sorted(set(r1.tolist()) | set(r2.tolist()))
    p_o = float(np.mean(r1 == r2))
    p_e = 0.0
    for lbl in labels:
        p1 = float(np.mean(r1 == lbl))
        p2 = float(np.mean(r2 == lbl))
        p_e += p1 * p2
    if abs(1.0 - p_e) < 1e-12:
        return 1.0 if abs(p_o - 1.0) < 1e-12 else 0.0
    return float((p_o - p_e) / (1.0 - p_e))

🧪 本地测试与正确性断言

可直接复制到本地 Python / Jupyter 验证:

import numpy as np
# 1. Position-bias detection: always picking first slot ('A' then 'A') must resolve to TIE
assert resolve_position_swap("A", "A") == "TIE", "Position bias must degrade to TIE"
assert resolve_position_swap("A", "B") == "A", "Consistent winner A must pass"
assert resolve_position_swap("B", "A") == "B", "Consistent winner B must pass"
# 2. Cohen's Kappa vs random chance agreement
human = ["A", "A", "B", "B", "A", "B"]
judge = ["A", "A", "B", "A", "A", "B"]
k = cohens_kappa(human, judge, labels=["A", "B"])
# Observed agreement = 5/6 = 0.8333, p_e = (3/6)*(4/6) + (3/6)*(2/6) = 0.5 -> kappa = (5/6 - 0.5)/0.5 = 0.6667
assert np.isclose(k, 2.0 / 3.0, atol=1e-4), f"Unexpected kappa: {k}"
print("✓ LLM-Judge swap & Cohen's Kappa assertion passed")

🎯 核心工程追问与标准应答

Q1:除了 Position Bias,LLM-as-a-Judge 还有哪两类常见系统性偏差?工程上怎么缓解?
常见还有 Verbosity Bias(偏爱更长但未必更准的回答)和 Self-Enhancement Bias(偏爱同系列模型生成的文本)。缓解手段包括:按长度分桶检验胜率、在 Prompt 中明确惩罚无意义冗余、以及使用跨家族多裁判集成(Judge Jury)并对人工抽样集校准 κ\kappa。
Q2:成对比较(Pairwise)与单条打分(Pointwise 1–5 分)在回归门控里各自适合什么场景?
Pairwise 对细微版本差异更敏感(A vs B 谁更好),适合 Prompt/模型灰度对比;Pointwise 成本更低(O(N)O(N) 而非 O(N2)O(N^2))且便于跨时间追踪绝对质量水位,常搭配可验证 Checklist Rubric 降低打分漂移。
⚡
想检验本题的肌肉记忆?进入 3 阶闪卡主动回忆盲测
隐藏答案遮罩,自测:审题锚点 ➔ 数理公式机理 ➔ 工业标杆代码与连环追问。3 分钟攻破认知盲区!
🎯 立即在闪卡中盲测此题➔
← 上一题: 滑动平均(Moving Average 窗口滑动计算)返回 Part M 与全部 105 题下一题: FSM 状态机结构化约束解码(Logit Masking) →