M1-054M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Medium
⚡Flashcard Drill➔Mastery:

Explain CUPED (Controlled-experiment Using Pre-Experiment Data) and Why It Achieves Massive Variance Reduction

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-055 →
📐 Mathematical Definition
Ycv=Y−θ(X−Xˉ),θ=Cov(Y,X)Var(X)Y^{cv}=Y-\theta(X-\bar X),\qquad \theta=\frac{\mathrm{Cov}(Y,X)}{\mathrm{Var}(X)}
⚡ One-Sentence Core Answer
CUPED removes predictable pre-experiment user variance by subtracting linear projections: Y∗=Y−θ(X−E[X])Y^* = Y - \theta (X - E[X]), reducing metric variance by factor (1−ρ2)(1 - \rho^2) and slashing required sample sizes.

📌 Key Takeaways

  • •
    Core Mechanism: Uses historical pre-experiment data XX as a control variate; since E[X−E[X]]=0E[X - E[X]] = 0, the treatment effect estimate remains completely unbiased.
  • •
    Optimal coefficient: θ∗=Cov(Y,X)Var(X)\theta^* = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}; identical to the ordinary least squares slope of YY on XX.
  • •
    Variance reduction factor: Var(Y∗)=Var(Y)(1−ρ2)\text{Var}(Y^*) = \text{Var}(Y)(1 - \rho^2), where ρ=Corr(Y,X)\rho = \text{Corr}(Y, X).

📐 Mathematical Derivations

Let YY be the experiment metric and XX be a pre-experiment metric measured before user exposure (hence strictly independent of treatment assignment: E[XA]=E[XB]=E[X]E[X_A] = E[X_B] = E[X]). Define adjusted metric Y∗=Y−θ(X−E[X])Y^* = Y - \theta(X - E[X]). The treatment effect estimate is Δ^CUPED=YˉB∗−YˉA∗=(YˉB−YˉA)−θ(XˉB−XˉA)\hat{\Delta}_{\text{CUPED}} = \bar{Y}_B^* - \bar{Y}_A^* = (\bar{Y}_B - \bar{Y}_A) - \theta(\bar{X}_B - \bar{X}_A). Taking expectations: E[Δ^CUPED]=E[YˉB−YˉA]−θ(E[X]−E[X])=ΔE[\hat{\Delta}_{\text{CUPED}}] = E[\bar{Y}_B - \bar{Y}_A] - \theta(E[X] - E[X]) = \Delta, preserving exact unbiasedness. The variance is Var(Y∗)=Var(Y)+θ2Var(X)−2θCov(Y,X)\text{Var}(Y^*) = \text{Var}(Y) + \theta^2\text{Var}(X) - 2\theta\text{Cov}(Y, X). Differentiating with respect to θ\theta and setting to 0 gives optimal θ∗=Cov(Y,X)Var(X)\theta^* = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}. Substituting θ∗\theta^* back: Var(Y∗)=Var(Y)−Cov(Y,X)2Var(X)=Var(Y)(1−Cov(Y,X)2Var(Y)Var(X))=Var(Y)(1−ρ2)\text{Var}(Y^*) = \text{Var}(Y) - \frac{\text{Cov}(Y, X)^2}{\text{Var}(X)} = \text{Var}(Y)\left(1 - \frac{\text{Cov}(Y, X)^2}{\text{Var}(Y)\text{Var}(X)}\right) = \text{Var}(Y)(1 - \rho^2).

🏭 Production Trade-offs

Industrial impact: In e-commerce and streaming where user engagement is highly stable (e.g. Past 2-week spend predicts future 2-week spend with ρ≈0.7\rho \approx 0.7), CUPED variance reduction is 1−0.72=51%1 - 0.7^2 = 51\%. This cuts the required experiment duration and sample size in half! If a user is brand new (cold start with no pre-experiment history), set Xi=XˉX_i = \bar{X} so Xi−Xˉ=0X_i - \bar{X} = 0, gracefully falling back to unadjusted YiY_i.
⚠️ Common Interview Pitfalls
  • ✕
    Using covariates XX that are measured after the user received treatment (which can be influenced by treatment, causing post-treatment conditioning bias).
  • ✕
    Computing θ\theta separately for Treatment and Control groups (must compute a pooled θ\theta across both to preserve exact unbiasedness).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is CUPED mathematically equivalent to ANCOVA (Analysis of Covariance) linear regression Y=α+βT+γX+ϵY = \alpha + \beta T + \gamma X + \epsilon?
  • ?
    How does CUPAC (CUPED with Machine Learning Predictions) generalize pre-experiment feature sets using Gradient Boosted Trees?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-053: Define Sample Ratio Mismatch (SRM), How to Detect It via Chi-Square, and Root Cause Diagnostics📋 Back to BankNext →M1-055: Analyze the Statistical Diagnostics When CTR Increases but CVR Remains Flat in an Online Experiment