M1-042M1: Mathematics & Statistics FundamentalsHypothesis TestingEasy
⚡Flashcard Drill➔Mastery:
Define Type I Error, Type II Error, Significance Level (Alpha), and Statistical Power (Beta)
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Type I error is false positive (rejecting true ); Type II error is false negative (failing to reject false ); Power is the probability of correctly detecting a real effect.
📌 Key Takeaways
- •Type I Error (): ; standard industrial threshold is .
- •Type II Error (): ; standard industrial threshold is .
- •Statistical Power (): Probability of detecting a true effect; standard target is or .
- •Fundamental tradeoff: Lowering shifts critical threshold outward, automatically increasing unless sample size is increased.
📐 Mathematical Derivations
For testing vs with test statistic : Rejection region is . Under , , giving Type I error . Under , . The Type II error is . Setting this to yields the fundamental relationship: , linking significance level, power, effect size, and sample size.
🏭 Production Trade-offs
In online experimentation: (1) If launching a risky feature (e.g. Redesigning core checkout UI), Type I error is dangerous, so one might tighten . (2) If screening hundreds of candidate algorithms in early funnel stages, Type II error is costlier (missing a breakthrough), so one might tolerate to boost power. The only way to decrease both and simultaneously is to increase sample size or reduce metric variance via CUPED.
⚠️ Common Interview Pitfalls
- ✕Concluding an experiment was successful simply because it had a high statistical power target (power is a pre-experiment planning metric, not a post-hoc evaluation).
- ✕Calculating 'post-hoc observed power' using the observed sample effect size (which is mathematically redundant with the p-value and provides zero new information).
🎯 Interviewer Follow-up Inquiries
- ?Why is post-hoc power considered mathematically invalid and misleading in experimental analysis?
- ?How does the Neyman-Pearson Lemma prove that likelihood ratio tests maximize power for simple hypotheses?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚