M1-050M1: Mathematics & Statistics FundamentalsConfidence Intervals & BootstrapHard
⚡Flashcard Drill➔Mastery:
Detail How to Apply Bootstrap to Evaluate Statistical Significance in A/B Testing and Identify Key Pitfalls
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
In A/B testing, bootstrap resamples users within treatment and control independently, constructing a confidence interval for difference ; statistical significance is established if zero falls outside the interval.
📌 Key Takeaways
- •Stratified resampling: Must resample Treatment and Control groups independently to preserve respective sample sizes and .
- •Significance criterion: Reject at level if .
- •Primary advantage: Seamlessly handles non-normal, ratio, and quantile metrics (P95 latency, Revenue per Converter).
📐 Mathematical Derivations
Algorithm for two-sample A/B bootstrap: Given control sample and treatment sample : (1) For : Resample points with replacement from to form , and resample points with replacement from to form . (2) Compute the metric of interest for both groups: and . (3) Compute the difference . (4) Construct the empirical percentile interval . If is not contained within the interval, the treatment effect is statistically significant at significance level .
🏭 Production Trade-offs
Common production pitfalls: (1) Wrong randomization unit: Resampling page views or sessions when users were randomized produces overly narrow confidence intervals (massive false positives due to intra-user correlation). Resampling must strictly occur at the User Level (Cluster Bootstrap). (2) Computational scaling: On billions of rows, running standard bootstrap is cost-prohibitive; platforms use Delta Method for ratio metrics or Poisson Bootstrap on pre-aggregated user stats.
⚠️ Common Interview Pitfalls
- ✕Pooling data and resampling globally across groups, which forces sample sizes to fluctuate randomly.
- ✕Resampling at the impression/click level instead of the user level, violating the i.i.d. assumption and severely inflating Type I error.
🎯 Interviewer Follow-up Inquiries
- ?How does cluster bootstrap resample at the user level when evaluating session-level or query-level metrics?
- ?Why does the Delta Method provide an instantaneous analytical alternative to Bootstrap for ratio metrics?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚