M1-054M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Medium
⚡Flashcard Drill➔Mastery:
Explain CUPED (Controlled-experiment Using Pre-Experiment Data) and Why It Achieves Massive Variance Reduction
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
CUPED removes predictable pre-experiment user variance by subtracting linear projections: , reducing metric variance by factor and slashing required sample sizes.
📌 Key Takeaways
- •Core Mechanism: Uses historical pre-experiment data as a control variate; since , the treatment effect estimate remains completely unbiased.
- •Optimal coefficient: ; identical to the ordinary least squares slope of on .
- •Variance reduction factor: , where .
📐 Mathematical Derivations
Let be the experiment metric and be a pre-experiment metric measured before user exposure (hence strictly independent of treatment assignment: ). Define adjusted metric . The treatment effect estimate is . Taking expectations: , preserving exact unbiasedness. The variance is . Differentiating with respect to and setting to 0 gives optimal . Substituting back: .
🏭 Production Trade-offs
Industrial impact: In e-commerce and streaming where user engagement is highly stable (e.g. Past 2-week spend predicts future 2-week spend with ), CUPED variance reduction is . This cuts the required experiment duration and sample size in half! If a user is brand new (cold start with no pre-experiment history), set so , gracefully falling back to unadjusted .
⚠️ Common Interview Pitfalls
- ✕Using covariates that are measured after the user received treatment (which can be influenced by treatment, causing post-treatment conditioning bias).
- ✕Computing separately for Treatment and Control groups (must compute a pooled across both to preserve exact unbiasedness).
🎯 Interviewer Follow-up Inquiries
- ?Why is CUPED mathematically equivalent to ANCOVA (Analysis of Covariance) linear regression ?
- ?How does CUPAC (CUPED with Machine Learning Predictions) generalize pre-experiment feature sets using Gradient Boosted Trees?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚