M1-052M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Easy
⚡Flashcard Drill➔Mastery:

Explain the Purpose and Diagnostic Criteria of A/A Testing in Online Experimentation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-053 →
📐 Mathematical Definition
A/A: Pr⁡(false positive)≈α\text{A/A}:\ \Pr(\text{false positive})\approx\alpha
⚡ One-Sentence Core Answer
An A/A test assigns identical experiences to both groups to validate experimental infrastructure integrity, verifying that the empirical false positive rate equals α\alpha and p-values follow a uniform distribution.

📌 Key Takeaways

  • •
    Primary Purpose: Validates traffic splitting randomization, pipeline data logging, and variance estimation algorithms without real treatment effects.
  • •
    Diagnostic Criterion 1: P-values across 1000 simulated A/A runs must be uniformly distributed U(0,1)\mathcal{U}(0, 1) (verified via Kolmogorov-Smirnov test).
  • •
    Diagnostic Criterion 2: Exactly α=5%\alpha = 5\% of A/A tests should reject H0H_0 at the 0.050.05 significance level; any higher rate signals pipeline bias.

📐 Mathematical Derivations

Under the sharp null hypothesis of identical variants (YA,YB∼FY_A, Y_B \sim F), the test statistic Z=YˉB−YˉAσA2/NA+σB2/NBZ = \frac{\bar{Y}_B - \bar{Y}_A}{\sqrt{\sigma_A^2/N_A + \sigma_B^2/N_B}} converges to standard normal N(0,1)\mathcal{N}(0, 1). The two-sided p-value is P=2(1−Φ(∣Z∣))P = 2(1 - \Phi(|Z|)). For any u∈[0,1]u \in [0, 1], P(p≤u)=P(2(1−Φ(∣Z∣))≤u)=P(∣Z∣≥Φ−1(1−u/2))=2(1−Φ(Φ−1(1−u/2)))=uP(p \le u) = P(2(1 - \Phi(|Z|)) \le u) = P(|Z| \ge \Phi^{-1}(1 - u/2)) = 2(1 - \Phi(\Phi^{-1}(1 - u/2))) = u, proving uniform distribution U(0,1)\mathcal{U}(0, 1). If the QQ-plot of empirical p-values deviates from the 45-degree diagonal line, or if the false positive rate is inflated, systematic pipeline errors exist.

🏭 Production Trade-offs

Modes of running A/A tests: (1) Dedicated A/A test: Run for 1-2 weeks before launching major new platform algorithms. (2) Continuous offline synthetic A/A testing: Re-split control group data into two random pseudo-buckets in historical logs and evaluate false positive rates across thousands of iterations, catching metric definition bugs with zero opportunity cost.
⚠️ Common Interview Pitfalls
  • ✕
    Discarding an A/B platform setup because a single A/A test yielded p=0.03p = 0.03 (by definition, 5% of all valid A/A tests yield p<0.05p < 0.05).
  • ✕
    Failing to check for Sample Ratio Mismatch (SRM) inside the A/A test.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Kolmogorov-Smirnov (KS) test formally test whether empirical A/A p-values deviate from U(0,1)\mathcal{U}(0, 1)?
  • ?
    Why does un-modeled intra-user correlation in session-level metrics cause severe A/A p-value inflation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-051: Describe the End-to-End Operational Lifecycle of a Standard Industrial A/B Experiment📋 Back to BankNext →M1-053: Define Sample Ratio Mismatch (SRM), How to Detect It via Chi-Square, and Root Cause Diagnostics