M1-079M1: Mathematics & Statistics FundamentalsHypothesis TestingHard
⚡Flashcard Drill➔Mastery:
Define Two One-Sided Tests (TOST) for Equivalence and How It Differs from Standard Null Hypothesis Testing
📐 Mathematical Definition
⚡ One-Sentence Core Answer
TOST reverses the burden of proof to demonstrate that two treatments are practically equivalent within an equivalence margin ; failure to reject difference in a standard t-test does NOT prove equivalence.
📌 Key Takeaways
- •Flaw of standard tests: In a standard -test (), failing to reject simply means under-powered data, NOT that groups are identical ('Absence of evidence is not evidence of absence').
- •TOST formulation: Sets non-equivalence as the null hypothesis: and .
- •Equivalence Decision: Conclude equivalence if and only if BOTH one-sided nulls are rejected at significance level (or equivalently, the confidence interval falls entirely within ).
📐 Mathematical Derivations
Mathematical setup: Define equivalence zone based on business tolerance (e.g. Latency change within ). TOST conducts two simultaneous one-sided tests: (1) . Reject if (proving effect is ). (2) . Reject if (proving effect is ). If both tests reject at level , we conclude with statistical confidence that the true difference is bounded strictly inside . Geometrically, this is identical to verifying that the two-sided confidence interval satisfies and .
🏭 Production Trade-offs
Essential for 'do-no-harm' migration experiments: (1) Upgrading infrastructure (e.g. Migrating from TensorFlow to PyTorch or refactoring a search retrieval service) where the goal is proving user engagement does not drop. (2) Cost reduction experiments: Shrinking LLM context or quantizing weights from FP16 to INT4, verifying that model accuracy remains within an equivalence margin .
⚠️ Common Interview Pitfalls
- ✕Running a standard A/B test with 500 users, observing , and claiming 'we proved the new system has identical performance' (the test was simply massively under-powered).
- ✕Setting the equivalence bound post-hoc after seeing data.
🎯 Interviewer Follow-up Inquiries
- ?Why does a confidence interval correspond to two simultaneous -level one-sided tests (TOST)?
- ?How does Non-Inferiority testing differ from Two One-Sided Equivalence Testing?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚
Associated Knowledge Base Guides & Mindmaps
Explore the comprehensive technical article, exam cards, and global architecture tree.
中文速览与面试要点
什么是等价性检验(TOST)?它与常规显著性检验有何不同?
TOST 检验'差异是否在等效边界内',用于证明'无实质差异';常规检验只能'未能拒绝无差异'。
核心要点
- 问题的根源:常规显著性检验的 H₀ 是'无差异', 未能拒绝 H₀ 只说明'证据不足',不能证明 H₀ 为真(可能是功效不足)。因此'A/B 测试不显著'不能推出'新方案与旧方案等效'——这正是很多团队误判的地方。 TOST(Two One-Sided Tests) 通过 反转假设 解决:H₀ 是'差异超出等效边界 |μ₁−μ₂|≥Δ',H₁ 是'差异在边界内'。做法是做两次单侧检验:① 检验 μ₁−μ₂ −Δ(下界);两次都拒绝才判定 等价 。等效边界 Δ 必须 预先由业务或临床意义确定 (如'转化率差异 等价性检验的置信区间视角 :(1−2α) 置信区间完全落在 [−Δ, Δ] 内即判等价。
- 应用场景与要点:① A/B 测试中的'无差异'结论 ——当新方案不显著且想上线(如降低成本的改动),应用 TOST 证明'没有实质变差';反之若想证明'新方案更差'则用常规检验。② 法规与临床 ——生物等效性(仿制药)与临床等效性研究 强制 使用 TOST(FDA/EMA 指南);Δ 由监管机构指定(如生物等效性用 80%–125% 的比值范围)。③ 功效分析的反转 ——TOST 的功效取决于 Δ、样本量与真实差异;若真实差异接近 0,所需样本量与常规检验相近;若真实差异接近 Δ,则需极大样本。④ 与'非劣效性检验'的区别 ——非劣效性只检验单侧(新方案不比旧方案差超过 Δ),是 TOST 的单边版本;常用于'新方案更便宜/更方便,只要不比旧的差太多'。⑤ 常见误用 ——(a) 把'不显著'当作等价(缺 TOST);(b) 事后选 Δ 使结论成立(应预注册);(c) 忽略功效——低功效的 TOST 也无法证明等价。⑥ 贝叶斯替代 ——用贝叶斯因子或后验分布(计算 P(|μ₁−μ₂|<Δ))也能给出等价性证据,且更直观。
- 两次单侧检验(下界与上界),都拒绝才判等价
- 必须预先设定等效边界 Δ(由业务/临床意义决定)
常见陷阱
- 把'不显著'当作'无差异'的证据(应用 TOST)
- 事后根据数据选等效边界 Δ