M1-049M1: Mathematics & Statistics FundamentalsConfidence Intervals & BootstrapMedium
⚡Flashcard Drill➔Mastery:

Contrast Bootstrap vs. Permutation Tests and Detail Their Respective Valid Assumptions and Use Cases

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-050 →
📐 Mathematical Definition
permutation:shuffle labels⇒null distribution\text{permutation}: \text{shuffle labels} \Rightarrow \text{null distribution}
⚡ One-Sentence Core Answer
Bootstrap resamples with replacement to estimate estimation uncertainty (confidence intervals); Permutation tests shuffle group labels without replacement to evaluate the sharp null hypothesis of identical distributions.

📌 Key Takeaways

  • •
    Bootstrap: Resamples with replacement (NN from NN) under general population model; primary goal is Confidence Intervals and standard errors.
  • •
    Permutation test: Shuffles labels without replacement under exchangeability assumption; primary goal is Exact P-values for hypothesis testing.
  • •
    Exchangeability: Permutation strictly requires that under H0H_0, observations across treatment and control are identically distributed and swappable.

📐 Mathematical Derivations

Permutation test mechanics: Testing H0:FA=FBH_0: F_A = F_B against H1:FA≠FBH_1: F_A \ne F_B with observed statistic Tobs=XˉB−XˉAT_{\text{obs}} = \bar{X}_B - \bar{X}_A. Under H0H_0, group labels ('Treatment' vs 'Control') are arbitrary. Pool all NA+NBN_A + N_B observations together. For p=1,…,Pp = 1, \dots, P: randomly shuffle the combined labels, reassign NAN_A points to pseudo-group AA and NBN_B points to pseudo-group BB, and compute T∗p=XˉB∗p−XˉA∗pT^{*p} = \bar{X}_B^{*p} - \bar{X}_A^{*p}. The exact two-sided p-value is p=1+∑p=1PI(∣T∗p∣≥∣Tobs∣)P+1p = \frac{1 + \sum_{p=1}^P \mathbb{I}(|T^{*p}| \ge |T_{\text{obs}}|)}{P + 1}. The added '1' in numerator and denominator ensures valid, non-zero exact frequentist type I error control.

🏭 Production Trade-offs

Practical selection guidelines: (1) Use Permutation Tests when sample size is small, distribution is completely unknown/skewed, and you require an exact, distribution-free p-value testing whether treatment has zero effect. (2) Use Bootstrap when you need to construct confidence intervals around treatment effect Δ\Delta, quantify estimation uncertainty, or compute ratios/quantiles.
⚠️ Common Interview Pitfalls
  • ✕
    Using permutation tests when group variances differ under the null hypothesis (Behrens-Fisher problem invalidates exchangeability, causing inflated false positives).
  • ✕
    Using sampling with replacement in a permutation test (which turns it into a bootstrap test).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the permutation test provide an exact finite-sample test while bootstrap provides only asymptotic guarantees?
  • ?
    How can permutation tests be applied to test independence between two continuous variables?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-048: Compare the Three Standard Bootstrap Interval Construction Methods (Percentile, Normal, and BCa Intervals)📋 Back to BankNext →M1-050: Detail How to Apply Bootstrap to Evaluate Statistical Significance in A/B Testing and Identify Key Pitfalls