M1-045M1: Mathematics & Statistics FundamentalsHypothesis TestingHard
⚡Flashcard Drill➔Mastery:

Explain Sequential Testing and the Peeking Problem, and Detail Correct Continuous Monitoring Frameworks

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-046 →
📐 Mathematical Definition
alpha spending: α(t) 随累计信息量分配,∑α(t)=α\text{alpha spending}:\ \alpha(t)\ \text{随累计信息量分配,}\sum\alpha(t)=\alpha
⚡ One-Sentence Core Answer
The Peeking Problem occurs when continuously checking p-values and stopping once p<0.05p<0.05, which inflates false positive rates from 5% to over 30%; resolved via alpha spending or sequential probability ratio tests.

📌 Key Takeaways

  • •
    The Peeking Problem: Repeatedly evaluating fixed-horizon tests over time turns the decision into sup⁡t≤TZt\sup_{t \le T} Z_t, guaranteeing eventual false positive crossing by the Law of Iterated Logarithm.
  • •
    Alpha Spending Function (Lan-DeMets / Pocock / O'Brien-Fleming): Allocates portions of total significance budget α\alpha across intermediate evaluation points.
  • •
    Always-Valid Inference (mSPRT / e-values): Constructs nonnegative supermartingales allowing arbitrary, continuous real-time peeking without inflating Type I error.

📐 Mathematical Derivations

Law of the Iterated Logarithm (LIL): For standardized random walk Sn=∑i=1nXiS_n = \sum_{i=1}^n X_i, lim sup⁡n→∞Sn2nlog⁡log⁡n=1\limsup_{n\to\infty} \frac{S_n}{\sqrt{2n \log \log n}} = 1 almost surely under H0H_0. Because 2nlog⁡log⁡n→∞\sqrt{2n \log \log n} \to \infty, the sample path will cross any fixed threshold z1−α/2nz_{1-\alpha/2} \sqrt{n} with probability 1 as n→∞n\to\infty. Hence, peeking daily for 30 days inflates effective α\alpha to ≈30%\approx 30\%. The Mixture Sequential Probability Ratio Test (mSPRT, Johari et al., 2017) computes test statistic Λn=∫∏i=1np(xi∣θ)p(xi∣0)dH(θ)\Lambda_n = \int \prod_{i=1}^n \frac{p(x_i\mid \theta)}{p(x_i\mid 0)} dH(\theta). By Ville's inequality for supermartingales, P(∃n:Λn≥1/α∣H0)≤αP(\exists n : \Lambda_n \ge 1/\alpha \mid H_0) \le \alpha, guaranteeing valid inference at all stopping times.

🏭 Production Trade-offs

Standard fixed-horizon A/B testing forbids inspecting results before reaching the planned sample size NN. However, product teams want to stop losing experiments early to minimize revenue destruction. Modern platforms (Optimizely, Netflix, Uber) implement mSPRT or O'Brien-Fleming boundaries, enabling continuous monitoring dashboards with rigorous statistical guarantees at the cost of slightly higher average sample size.
⚠️ Common Interview Pitfalls
  • ✕
    Checking the A/B dashboard every morning and turning off the experiment the moment p<0.05p < 0.05.
  • ✕
    Treating standard confidence intervals as valid when early stopping rules were applied.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Ville's inequality for nonnegative supermartingales provide time-uniform confidence boundaries?
  • ?
    What is the difference between Pocock boundaries (constant threshold) and O'Brien-Fleming boundaries (conservative early on)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-044: Define the Multiple Testing Problem and Contrast Common Corrections (Bonferroni vs. Benjamini-Hochberg FDR)📋 Back to BankNext →M1-046: Explain the Frequentist Interpretation of Confidence Intervals and Their Duality with Hypothesis Testing