M1-079M1: Mathematics & Statistics FundamentalsHypothesis TestingHard
⚡Flashcard Drill➔Mastery:

Define Two One-Sided Tests (TOST) for Equivalence and How It Differs from Standard Null Hypothesis Testing

📐 Mathematical Definition
TOST: H0: ∣μ1−μ2∣≥ΔvsH1: ∣μ1−μ2∣<Δ\text{TOST}:\ H_0:\ |\mu_1-\mu_2|\ge\Delta\quad\text{vs}\quad H_1:\ |\mu_1-\mu_2|<\Delta
⚡ One-Sentence Core Answer
TOST reverses the burden of proof to demonstrate that two treatments are practically equivalent within an equivalence margin [−Δ,+Δ][-\Delta, +\Delta]; failure to reject difference in a standard t-test does NOT prove equivalence.

📌 Key Takeaways

  • •
    Flaw of standard tests: In a standard tt-test (H0:μT−μC=0H_0: \mu_T - \mu_C = 0), failing to reject p>0.05p > 0.05 simply means under-powered data, NOT that groups are identical ('Absence of evidence is not evidence of absence').
  • •
    TOST formulation: Sets non-equivalence as the null hypothesis: H01:μT−μC≤−ΔH_{01}: \mu_T - \mu_C \le -\Delta and H02:μT−μC≥+ΔH_{02}: \mu_T - \mu_C \ge +\Delta.
  • •
    Equivalence Decision: Conclude equivalence if and only if BOTH one-sided nulls are rejected at significance level α\alpha (or equivalently, the 90%90\% confidence interval falls entirely within [−Δ,+Δ][-\Delta, +\Delta]).

📐 Mathematical Derivations

Mathematical setup: Define equivalence zone [−Δ,+Δ][-\Delta, +\Delta] based on business tolerance (e.g. Latency change within ±5ms\pm 5\text{ms}). TOST conducts two simultaneous one-sided tests: (1) t1=(YˉT−YˉC)−(−Δ)set_1 = \frac{(\bar{Y}_T - \bar{Y}_C) - (-\Delta)}{\text{se}}. Reject H01H_{01} if t1>t1−α,νt_1 > t_{1-\alpha, \nu} (proving effect is >−Δ> -\Delta). (2) t2=(YˉT−YˉC)−Δset_2 = \frac{(\bar{Y}_T - \bar{Y}_C) - \Delta}{\text{se}}. Reject H02H_{02} if t2<−t1−α,νt_2 < -t_{1-\alpha, \nu} (proving effect is <+Δ< +\Delta). If both tests reject at level α\alpha, we conclude with statistical confidence 1−2α1 - 2\alpha that the true difference is bounded strictly inside (−Δ,Δ)(-\Delta, \Delta). Geometrically, this is identical to verifying that the (1−2α)(1 - 2\alpha) two-sided confidence interval [L,U][L, U] satisfies −Δ<L-\Delta < L and U<ΔU < \Delta.

🏭 Production Trade-offs

Essential for 'do-no-harm' migration experiments: (1) Upgrading infrastructure (e.g. Migrating from TensorFlow to PyTorch or refactoring a search retrieval service) where the goal is proving user engagement does not drop. (2) Cost reduction experiments: Shrinking LLM context or quantizing weights from FP16 to INT4, verifying that model accuracy remains within an equivalence margin Δ=0.5%\Delta = 0.5\%.
⚠️ Common Interview Pitfalls
  • ✕
    Running a standard A/B test with 500 users, observing p=0.40p = 0.40, and claiming 'we proved the new system has identical performance' (the test was simply massively under-powered).
  • ✕
    Setting the equivalence bound Δ\Delta post-hoc after seeing data.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does a (1−2α)(1 - 2\alpha) confidence interval correspond to two simultaneous α\alpha-level one-sided tests (TOST)?
  • ?
    How does Non-Inferiority testing differ from Two One-Sided Equivalence Testing?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

Explore the comprehensive technical article, exam cards, and global architecture tree.

← PreviousM1-078: Explain the Wilcoxon Rank-Sum Test and When Non-Parametric Rank Tests Should Be Chosen📋Back to BankNext →M1-080: Explain Bayes Factors and the Bayesian Perspective on Hypothesis Testing
中文速览与面试要点

什么是等价性检验(TOST)?它与常规显著性检验有何不同?

TOST 检验'差异是否在等效边界内',用于证明'无实质差异';常规检验只能'未能拒绝无差异'。

核心要点

  • 问题的根源:常规显著性检验的 H₀ 是'无差异', 未能拒绝 H₀ 只说明'证据不足',不能证明 H₀ 为真(可能是功效不足)。因此'A/B 测试不显著'不能推出'新方案与旧方案等效'——这正是很多团队误判的地方。 TOST(Two One-Sided Tests) 通过 反转假设 解决:H₀ 是'差异超出等效边界 |μ₁−μ₂|≥Δ',H₁ 是'差异在边界内'。做法是做两次单侧检验:① 检验 μ₁−μ₂ −Δ(下界);两次都拒绝才判定 等价 。等效边界 Δ 必须 预先由业务或临床意义确定 (如'转化率差异 等价性检验的置信区间视角 :(1−2α) 置信区间完全落在 [−Δ, Δ] 内即判等价。
  • 应用场景与要点:① A/B 测试中的'无差异'结论 ——当新方案不显著且想上线(如降低成本的改动),应用 TOST 证明'没有实质变差';反之若想证明'新方案更差'则用常规检验。② 法规与临床 ——生物等效性(仿制药)与临床等效性研究 强制 使用 TOST(FDA/EMA 指南);Δ 由监管机构指定(如生物等效性用 80%–125% 的比值范围)。③ 功效分析的反转 ——TOST 的功效取决于 Δ、样本量与真实差异;若真实差异接近 0,所需样本量与常规检验相近;若真实差异接近 Δ,则需极大样本。④ 与'非劣效性检验'的区别 ——非劣效性只检验单侧(新方案不比旧方案差超过 Δ),是 TOST 的单边版本;常用于'新方案更便宜/更方便,只要不比旧的差太多'。⑤ 常见误用 ——(a) 把'不显著'当作等价(缺 TOST);(b) 事后选 Δ 使结论成立(应预注册);(c) 忽略功效——低功效的 TOST 也无法证明等价。⑥ 贝叶斯替代 ——用贝叶斯因子或后验分布(计算 P(|μ₁−μ₂|<Δ))也能给出等价性证据,且更直观。
  • 两次单侧检验(下界与上界),都拒绝才判等价
  • 必须预先设定等效边界 Δ(由业务/临床意义决定)

常见陷阱

  • 把'不显著'当作'无差异'的证据(应用 TOST)
  • 事后根据数据选等效边界 Δ