M1-051M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Easy
⚡Flashcard Drill➔Mastery:
Describe the End-to-End Operational Lifecycle of a Standard Industrial A/B Experiment
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
A standard A/B experiment progresses through seven rigorous phases: business hypothesis formulation, metric taxonomy definition, power-based sample sizing, randomization hashing, SRM integrity sanity checks, significance evaluation, and post-launch guardrail monitoring.
📌 Key Takeaways
- •Lifecycle: 1. Hypothesis & Metrics -> 2. Sample Size (MDE) -> 3. Hash Assignment -> 4. A/A & SRM Checks -> 5. Statistical Inference -> 6. Decision & Rollout.
- •Metric Hierarchy: Primary decision metric (e.g. conversion rate), Secondary behavioral diagnostics (click depth), and Guardrail safety metrics (latency, crash rate, unsubscribe rate).
- •Unit of Randomization: Must align strictly with the treatment experience (e.g. User ID for personalized UI, Request ID for stateless backend optimizations).
📐 Mathematical Derivations
Rigorous statistical protocol: (1) Sizing: Compute minimum sample size per variant for targeted MDE . (2) Assignment: Hash user identifier using salt: . (3) Integrity Check: Evaluate Sample Ratio Mismatch via Chi-Square goodness-of-fit: . If , the experiment is invalid. (4) Treatment Effect: Compute lift with standard error , applying CUPED covariate adjustment.
🏭 Production Trade-offs
Common industrial trade-offs: (1) Duration vs. Novelty effect: Running for at least 1-2 full weeks captures weekly cyclical patterns (weekend vs weekday) while letting novelty effects (users clicking due to unfamiliar UI) burn off. (2) Ramp-up strategy: Start at 1% -> 5% -> 50% traffic allocation to protect production revenue against catastrophic bugs before launching full statistical evaluation.
⚠️ Common Interview Pitfalls
- ✕Changing the allocation ratio mid-experiment without stratifying analysis (causes severe Simpson's paradox and artificial SRM).
- ✕Concluding significance based on secondary metrics when the pre-registered primary metric was flat (cherry-picking).
🎯 Interviewer Follow-up Inquiries
- ?How do multiple layers of concurrent experiments run without interference using orthogonal hashing salts?
- ?What is the difference between intent-to-treat (ITT) and average treatment effect on the treated (ATT) in online experiments?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚