M1-052M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Easy
⚡Flashcard Drill➔Mastery:
Explain the Purpose and Diagnostic Criteria of A/A Testing in Online Experimentation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
An A/A test assigns identical experiences to both groups to validate experimental infrastructure integrity, verifying that the empirical false positive rate equals and p-values follow a uniform distribution.
📌 Key Takeaways
- •Primary Purpose: Validates traffic splitting randomization, pipeline data logging, and variance estimation algorithms without real treatment effects.
- •Diagnostic Criterion 1: P-values across 1000 simulated A/A runs must be uniformly distributed (verified via Kolmogorov-Smirnov test).
- •Diagnostic Criterion 2: Exactly of A/A tests should reject at the significance level; any higher rate signals pipeline bias.
📐 Mathematical Derivations
Under the sharp null hypothesis of identical variants (), the test statistic converges to standard normal . The two-sided p-value is . For any , , proving uniform distribution . If the QQ-plot of empirical p-values deviates from the 45-degree diagonal line, or if the false positive rate is inflated, systematic pipeline errors exist.
🏭 Production Trade-offs
Modes of running A/A tests: (1) Dedicated A/A test: Run for 1-2 weeks before launching major new platform algorithms. (2) Continuous offline synthetic A/A testing: Re-split control group data into two random pseudo-buckets in historical logs and evaluate false positive rates across thousands of iterations, catching metric definition bugs with zero opportunity cost.
⚠️ Common Interview Pitfalls
- ✕Discarding an A/B platform setup because a single A/A test yielded (by definition, 5% of all valid A/A tests yield ).
- ✕Failing to check for Sample Ratio Mismatch (SRM) inside the A/A test.
🎯 Interviewer Follow-up Inquiries
- ?How does the Kolmogorov-Smirnov (KS) test formally test whether empirical A/A p-values deviate from ?
- ?Why does un-modeled intra-user correlation in session-level metrics cause severe A/A p-value inflation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚