M7-007HardLLM 评估 · LLM evaluation
LLM-as-judge 有哪些已知偏差?如何验证 judge 可信?
What biases does LLM-as-judge have, and how do you validate the judge?
💡 考场高分一句话答案:
位置、长度、自我偏好、格式四类偏差;用人工标注样本算一致率(κ / Spearman)来验证,低于 0.6 不可用于决策。
Position, length, self-preference and format bias. Validate against 200 to 500 human labels and require agreement above roughly 0.6 before using it for decisions.
📌 核心考察要点 (Key Points)
- 位置偏差:随机交换 A/B 顺序,取两次平均
- 长度偏差:控制长度,或把长度作为协变量回归掉
- 自我偏好:用不同家族模型做 judge,或多 judge 投票
- 验证:200–500 条人工标注,算与人的一致率;分维度看(正确性通常一致率高,有用性低)
- Position bias: randomise the order of A and B and average both directions
- Length bias: control for length, or regress it out as a covariate
- Self-preference: use a different model family as judge, or majority-vote across judges
- Validation: compute agreement per dimension; factual correctness usually agrees better than usefulness
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
judge 的可靠性还依赖任务难度:在“事实正确性”上 judge 与人一致性可达 0.8+,在“写作质量/有用性”上往往只有 0.4–0.5。因此实践做法是“硬指标用 judge,软指标用人”,而不是全面替代。
⚠️ 常见高频面试陷阱与避坑指南
- 用同一个模型既生成又评判(自我偏好放大)
- 不做顺序交换,直接比较 A/B 位置
🔍 大厂面试官连环追问 (Follow-ups)
- 如何设计一个 pairwise 评估协议?
- Bradley-Terry 模型如何聚合 pairwise 结果?
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.