M1-079M1: Mathematics & Statistics FundamentalsHypothesis TestingHard
⚡Flashcard Drill➔Mastery:

Define Two One-Sided Tests (TOST) for Equivalence and How It Differs from Standard Null Hypothesis Testing

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-080 →
📐 Mathematical Definition
TOST: H0: ∣μ1−μ2∣≥ΔvsH1: ∣μ1−μ2∣<Δ\text{TOST}:\ H_0:\ |\mu_1-\mu_2|\ge\Delta\quad\text{vs}\quad H_1:\ |\mu_1-\mu_2|<\Delta
⚡ One-Sentence Core Answer
TOST reverses the burden of proof to demonstrate that two treatments are practically equivalent within an equivalence margin [−Δ,+Δ][-\Delta, +\Delta]; failure to reject difference in a standard t-test does NOT prove equivalence.

📌 Key Takeaways

  • •
    Flaw of standard tests: In a standard tt-test (H0:μT−μC=0H_0: \mu_T - \mu_C = 0), failing to reject p>0.05p > 0.05 simply means under-powered data, NOT that groups are identical ('Absence of evidence is not evidence of absence').
  • •
    TOST formulation: Sets non-equivalence as the null hypothesis: H01:μT−μC≤−ΔH_{01}: \mu_T - \mu_C \le -\Delta and H02:μT−μC≥+ΔH_{02}: \mu_T - \mu_C \ge +\Delta.
  • •
    Equivalence Decision: Conclude equivalence if and only if BOTH one-sided nulls are rejected at significance level α\alpha (or equivalently, the 90%90\% confidence interval falls entirely within [−Δ,+Δ][-\Delta, +\Delta]).

📐 Mathematical Derivations

Mathematical setup: Define equivalence zone [−Δ,+Δ][-\Delta, +\Delta] based on business tolerance (e.g. Latency change within ±5ms\pm 5\text{ms}). TOST conducts two simultaneous one-sided tests: (1) t1=(YˉT−YˉC)−(−Δ)set_1 = \frac{(\bar{Y}_T - \bar{Y}_C) - (-\Delta)}{\text{se}}. Reject H01H_{01} if t1>t1−α,νt_1 > t_{1-\alpha, \nu} (proving effect is >−Δ> -\Delta). (2) t2=(YˉT−YˉC)−Δset_2 = \frac{(\bar{Y}_T - \bar{Y}_C) - \Delta}{\text{se}}. Reject H02H_{02} if t2<−t1−α,νt_2 < -t_{1-\alpha, \nu} (proving effect is <+Δ< +\Delta). If both tests reject at level α\alpha, we conclude with statistical confidence 1−2α1 - 2\alpha that the true difference is bounded strictly inside (−Δ,Δ)(-\Delta, \Delta). Geometrically, this is identical to verifying that the (1−2α)(1 - 2\alpha) two-sided confidence interval [L,U][L, U] satisfies −Δ<L-\Delta < L and U<ΔU < \Delta.

🏭 Production Trade-offs

Essential for 'do-no-harm' migration experiments: (1) Upgrading infrastructure (e.g. Migrating from TensorFlow to PyTorch or refactoring a search retrieval service) where the goal is proving user engagement does not drop. (2) Cost reduction experiments: Shrinking LLM context or quantizing weights from FP16 to INT4, verifying that model accuracy remains within an equivalence margin Δ=0.5%\Delta = 0.5\%.
⚠️ Common Interview Pitfalls
  • ✕
    Running a standard A/B test with 500 users, observing p=0.40p = 0.40, and claiming 'we proved the new system has identical performance' (the test was simply massively under-powered).
  • ✕
    Setting the equivalence bound Δ\Delta post-hoc after seeing data.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does a (1−2α)(1 - 2\alpha) confidence interval correspond to two simultaneous α\alpha-level one-sided tests (TOST)?
  • ?
    How does Non-Inferiority testing differ from Two One-Sided Equivalence Testing?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-078: Explain the Wilcoxon Rank-Sum Test and When Non-Parametric Rank Tests Should Be Chosen📋 Back to BankNext →M1-080: Explain Bayes Factors and the Bayesian Perspective on Hypothesis Testing