M2-014MediumAI 功能评估 · AI feature evaluation

如何评估一个 AI 客服/助手的“解决率”?光看解决率够吗?

How would you evaluate an AI support assistant's resolution rate? Is resolution rate enough?
💡 考场高分一句话答案:

不够。要区分“声称解决”与“真正解决”,并用 7 日重复联系率、CSAT、以及人工抽检作交叉验证。

No. Separate claimed resolution from actual resolution, and cross-check with 7-day repeat contact, CSAT and sampled human review.

📌 核心考察要点 (Key Points)

  • 定义:会话结束时用户未再联系 = 表面解决;7 日内未就同一问题联系 = 真实解决
  • 反向指标:重复联系率、升级到人工率、会话后投诉率
  • 质量抽检:按 1–5% 抽样人工评审“回答是否正确且完整”
  • 成本侧:每次解决的成本(token + 延迟 + 人工兜底)
  • Definitions: the user does not recontact within the session is claimed resolution; no repeat contact within 7 days on the same issue is actual resolution
  • Counter-metrics: repeat contact rate, escalation to human rate, complaint rate after the session
  • Quality sampling: 1–5% human review of whether the answer was correct and complete
  • Cost side: cost per resolved issue (tokens, latency, human fallback)

⚖️ 业务权衡与取舍 (Trade-offs & Calibration)

AI 客服最常见的度量陷阱是“用户放弃了”被算成“解决了”:用户不再联系可能因为失望而不是满意。判据是看“后续 30 天的整体联系率”与“NPS/CSAT”,如果解决率上升但整体联系率没降,说明只是把问题推后了。

⚠️ 常见高频面试陷阱与避坑指南

  • 用“会话正常结束”当作解决
  • 不做人工抽检,纯靠行为代理

🔍 大厂面试官连环追问 (Follow-ups)

  • 如何设计一个防作弊的解决率指标?
  • 如果解决率高但 CSAT 低,说明什么?

嫌字多看太累?进入苏格拉底式对话推演实验室

零打字负担,以大厂考官真实困境切入,通过点选直觉选项暴露思维盲区,手把手推导 Why。

💬 开启对话式破冰 ➔
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔