M7-009MediumLLM 评估 · LLM evaluation
如何度量 LLM 的“幻觉率”?
How do you measure an LLM's hallucination rate?
💡 考场高分一句话答案:
分类型度量:事实错误(可核查)、无据推断(上下文不支持)、引用错误(引用不存在或不支持论断),分别用不同方法。
Measure by type: factual errors, unsupported inference, and citation errors, each with its own method and reported with harm weighting rather than one average.
📌 核心考察要点 (Key Points)
- 事实错误:与知识库/检索结果比对,或用独立模型核查(需人工抽检校准)
- 无据推断:faithfulness——每个论断是否被上下文支持(NLI 模型或 judge)
- 引用错误:检查 URL 是否存在、内容是否支持该论断
- 报告方式:按危害加权(医疗/金融类错误权重更高),而不是单一平均率
- Factual errors: compare against a knowledge base or retrieved context, with human spot checks to calibrate
- Unsupported inference: faithfulness — whether each claim is entailed by the context
- Citation errors: does the URL exist and does it support the claim
- Reporting: weight by harm (medical, financial), and report 'should-answer-but-did-not' separately from 'answered wrongly'
⚖️ 业务权衡与取舍 (Trade-offs & Calibration)
幻觉率的分母很关键:把“我不知道”算成错误会激励模型乱答;不算又会低估。稳妥做法是分别报告“该答未答率”与“答错率”,并明确“拒答”是允许的行为。这与分类器的 precision/recall 权衡是同一个结构。
⚠️ 常见高频面试陷阱与避坑指南
- 用一个笼统的“幻觉率”掩盖类型差异
- 分母包含本就没有答案的问题
🔍 大厂面试官连环追问 (Follow-ups)
- 如何自动检测引用是否支持论断?
- 拒答率与幻觉率如何权衡?
嫌字多看太累?进入苏格拉底式对话推演实验室
零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.