M1-045M1: Mathematics & Statistics FundamentalsHypothesis TestingHard
⚡Flashcard Drill➔Mastery:
Explain Sequential Testing and the Peeking Problem, and Detail Correct Continuous Monitoring Frameworks
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
The Peeking Problem occurs when continuously checking p-values and stopping once , which inflates false positive rates from 5% to over 30%; resolved via alpha spending or sequential probability ratio tests.
📌 Key Takeaways
- •The Peeking Problem: Repeatedly evaluating fixed-horizon tests over time turns the decision into , guaranteeing eventual false positive crossing by the Law of Iterated Logarithm.
- •Alpha Spending Function (Lan-DeMets / Pocock / O'Brien-Fleming): Allocates portions of total significance budget across intermediate evaluation points.
- •Always-Valid Inference (mSPRT / e-values): Constructs nonnegative supermartingales allowing arbitrary, continuous real-time peeking without inflating Type I error.
📐 Mathematical Derivations
Law of the Iterated Logarithm (LIL): For standardized random walk , almost surely under . Because , the sample path will cross any fixed threshold with probability 1 as . Hence, peeking daily for 30 days inflates effective to . The Mixture Sequential Probability Ratio Test (mSPRT, Johari et al., 2017) computes test statistic . By Ville's inequality for supermartingales, , guaranteeing valid inference at all stopping times.
🏭 Production Trade-offs
Standard fixed-horizon A/B testing forbids inspecting results before reaching the planned sample size . However, product teams want to stop losing experiments early to minimize revenue destruction. Modern platforms (Optimizely, Netflix, Uber) implement mSPRT or O'Brien-Fleming boundaries, enabling continuous monitoring dashboards with rigorous statistical guarantees at the cost of slightly higher average sample size.
⚠️ Common Interview Pitfalls
- ✕Checking the A/B dashboard every morning and turning off the experiment the moment .
- ✕Treating standard confidence intervals as valid when early stopping rules were applied.
🎯 Interviewer Follow-up Inquiries
- ?How does Ville's inequality for nonnegative supermartingales provide time-uniform confidence boundaries?
- ?What is the difference between Pocock boundaries (constant threshold) and O'Brien-Fleming boundaries (conservative early on)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚