M1-043M1: Mathematics & Statistics FundamentalsHypothesis TestingMedium
⚡Flashcard Drill➔Mastery:

Explain the Mathematical Relationship Between Sample Size and Statistical Power with Sizing Formulas

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-044 →
📐 Mathematical Definition
n ∝ (z1−α/2+z1−β)2 σ2Δ2n\ \propto\ \frac{(z_{1-\alpha/2}+z_{1-\beta})^2\,\sigma^2}{\Delta^2}
⚡ One-Sentence Core Answer
Sample size scales inversely with the square of Minimum Detectable Effect (N∝1/δ2N \propto 1/\delta^2); halving the effect size requires quadrupling the sample size.

📌 Key Takeaways

  • •
    Standard sample size formula: N=2(z1−α/2+z1−β)2σ2δ2N = \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{\delta^2} for two-sample equal-allocation tests.
  • •
    Square-root law: Standard error shrinks at rate O(1/N)O(1/\sqrt{N}), compelling exponential sample requirements for tiny metrics.
  • •
    Key inputs: Significance level α\alpha (typically 0.05), Power 1−β1-\beta (typically 0.80), Baseline variance σ2\sigma^2, and MDE δ\delta.

📐 Mathematical Derivations

For two-sample z-test with NN users per variant, the difference in sample means Δ=XˉB−XˉA\Delta = \bar{X}_B - \bar{X}_A has variance Var(Δ)=σA2N+σB2N≈2σ2N\text{Var}(\Delta) = \frac{\sigma_A^2}{N} + \frac{\sigma_B^2}{N} \approx \frac{2\sigma^2}{N}. Under H0H_0, critical threshold is C=z1−α/22σ2NC = z_{1-\alpha/2} \sqrt{\frac{2\sigma^2}{N}}. Under H1:Δ=δH_1: \Delta = \delta, statistical power requires P(Δ>C∣H1)=1−βP(\Delta > C \mid H_1) = 1 - \beta, which means C=δ−z1−β2σ2NC = \delta - z_{1-\beta} \sqrt{\frac{2\sigma^2}{N}}. Equating the two expressions for CC: z1−α/22σ2N=δ−z1−β2σ2N  ⟹  (z1−α/2+z1−β)2σ2N=δz_{1-\alpha/2} \sqrt{\frac{2\sigma^2}{N}} = \delta - z_{1-\beta} \sqrt{\frac{2\sigma^2}{N}} \implies (z_{1-\alpha/2} + z_{1-\beta}) \sqrt{\frac{2\sigma^2}{N}} = \delta. Squaring and solving for NN: N=2(z1−α/2+z1−β)2σ2δ2N = \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{\delta^2}. For α=0.05,1−β=0.80\alpha=0.05, 1-\beta=0.80, (1.96+0.84)2=7.84(1.96 + 0.84)^2 = 7.84, giving rule-of-thumb N≈16σ2δ2N \approx \frac{16\sigma^2}{\delta^2}.

🏭 Production Trade-offs

If traffic is constrained and MDE cannot be achieved within acceptable timeframes, applied scientists have three options: (1) Variance reduction (CUPED): Reduces effective σ2\sigma^2 by factor (1−ρ2)(1-\rho^2), cutting required NN by up to 50%. (2) Transform metrics: Switch from noisy continuous revenue to conversion rate or binarized flags with lower coefficient of variation. (3) Multi-week run durations to capture weekly seasonality.
⚠️ Common Interview Pitfalls
  • ✕
    Setting MDE based on wishful thinking rather than business ROI thresholds.
  • ✕
    Ignoring clustering in sample size calculations when randomization unit differs from analysis unit (e.g. User-level treatment with page-level metrics).
🎯 Interviewer Follow-up Inquiries
  • ?
    How does sample size calculation change for binary proportion metrics (p(1−p)p(1-p) variance)?
  • ?
    Why must sample size be inflated by the Design Effect (DEFF) in cluster-randomized experiments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-042: Define Type I Error, Type II Error, Significance Level (Alpha), and Statistical Power (Beta)📋 Back to BankNext →M1-044: Define the Multiple Testing Problem and Contrast Common Corrections (Bonferroni vs. Benjamini-Hochberg FDR)