M1-043M1: Mathematics & Statistics FundamentalsHypothesis TestingMedium
⚡Flashcard Drill➔Mastery:
Explain the Mathematical Relationship Between Sample Size and Statistical Power with Sizing Formulas
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sample size scales inversely with the square of Minimum Detectable Effect (); halving the effect size requires quadrupling the sample size.
📌 Key Takeaways
- •Standard sample size formula: for two-sample equal-allocation tests.
- •Square-root law: Standard error shrinks at rate , compelling exponential sample requirements for tiny metrics.
- •Key inputs: Significance level (typically 0.05), Power (typically 0.80), Baseline variance , and MDE .
📐 Mathematical Derivations
For two-sample z-test with users per variant, the difference in sample means has variance . Under , critical threshold is . Under , statistical power requires , which means . Equating the two expressions for : . Squaring and solving for : . For , , giving rule-of-thumb .
🏭 Production Trade-offs
If traffic is constrained and MDE cannot be achieved within acceptable timeframes, applied scientists have three options: (1) Variance reduction (CUPED): Reduces effective by factor , cutting required by up to 50%. (2) Transform metrics: Switch from noisy continuous revenue to conversion rate or binarized flags with lower coefficient of variation. (3) Multi-week run durations to capture weekly seasonality.
⚠️ Common Interview Pitfalls
- ✕Setting MDE based on wishful thinking rather than business ROI thresholds.
- ✕Ignoring clustering in sample size calculations when randomization unit differs from analysis unit (e.g. User-level treatment with page-level metrics).
🎯 Interviewer Follow-up Inquiries
- ?How does sample size calculation change for binary proportion metrics ( variance)?
- ?Why must sample size be inflated by the Design Effect (DEFF) in cluster-randomized experiments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚