M1-051M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Easy
⚡Flashcard Drill➔Mastery:

Describe the End-to-End Operational Lifecycle of a Standard Industrial A/B Experiment

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-052 →
📐 Mathematical Definition
n∝σ2Δ2,SUTVA: 无干扰n\propto\frac{\sigma^2}{\Delta^2},\qquad \text{SUTVA}:\ \text{无干扰}
⚡ One-Sentence Core Answer
A standard A/B experiment progresses through seven rigorous phases: business hypothesis formulation, metric taxonomy definition, power-based sample sizing, randomization hashing, SRM integrity sanity checks, significance evaluation, and post-launch guardrail monitoring.

📌 Key Takeaways

  • •
    Lifecycle: 1. Hypothesis & Metrics -> 2. Sample Size (MDE) -> 3. Hash Assignment -> 4. A/A & SRM Checks -> 5. Statistical Inference -> 6. Decision & Rollout.
  • •
    Metric Hierarchy: Primary decision metric (e.g. conversion rate), Secondary behavioral diagnostics (click depth), and Guardrail safety metrics (latency, crash rate, unsubscribe rate).
  • •
    Unit of Randomization: Must align strictly with the treatment experience (e.g. User ID for personalized UI, Request ID for stateless backend optimizations).

📐 Mathematical Derivations

Rigorous statistical protocol: (1) Sizing: Compute minimum sample size per variant N=2(z1−α/2+z1−β)2σ2δ2N = \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{\delta^2} for targeted MDE δ\delta. (2) Assignment: Hash user identifier using salt: Bucket=MurmurHash3(user_id+salt)(mod100)\text{Bucket} = \text{MurmurHash3}(\text{user\_id} + \text{salt}) \pmod{100}. (3) Integrity Check: Evaluate Sample Ratio Mismatch via Chi-Square goodness-of-fit: χ2=∑i(Oi−Ei)2Ei∼χk−12\chi^2 = \sum_{i} \frac{(O_i - E_i)^2}{E_i} \sim \chi^2_{k-1}. If pSRM<0.001p_{\text{SRM}} < 0.001, the experiment is invalid. (4) Treatment Effect: Compute lift Δ^=YˉB−YˉA\hat{\Delta} = \bar{Y}_B - \bar{Y}_A with standard error se(Δ^)=sA2NA+sB2NB\text{se}(\hat{\Delta}) = \sqrt{\frac{s_A^2}{N_A} + \frac{s_B^2}{N_B}}, applying CUPED covariate adjustment.

🏭 Production Trade-offs

Common industrial trade-offs: (1) Duration vs. Novelty effect: Running for at least 1-2 full weeks captures weekly cyclical patterns (weekend vs weekday) while letting novelty effects (users clicking due to unfamiliar UI) burn off. (2) Ramp-up strategy: Start at 1% -> 5% -> 50% traffic allocation to protect production revenue against catastrophic bugs before launching full statistical evaluation.
⚠️ Common Interview Pitfalls
  • ✕
    Changing the allocation ratio mid-experiment without stratifying analysis (causes severe Simpson's paradox and artificial SRM).
  • ✕
    Concluding significance based on secondary metrics when the pre-registered primary metric was flat (cherry-picking).
🎯 Interviewer Follow-up Inquiries
  • ?
    How do multiple layers of concurrent experiments run without interference using orthogonal hashing salts?
  • ?
    What is the difference between intent-to-treat (ITT) and average treatment effect on the treated (ATT) in online experiments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-050: Detail How to Apply Bootstrap to Evaluate Statistical Significance in A/B Testing and Identify Key Pitfalls📋 Back to BankNext →M1-052: Explain the Purpose and Diagnostic Criteria of A/A Testing in Online Experimentation