O3Medium★ Experimentation & RL Control Variate BridgePart O · BA-to-Python Data Science & Stats Kernels

CUPED A/B Variance Reduction with Pooled Covariate Control

⚡Flashcard Drill➔

Implement unbiased CUPED estimation for A/B experiments and see how reducing variance by rho^2 mirrors baseline subtraction in RL policy gradients.

⏱️ Time: O(N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def cuped_ab_evaluation(y: np.ndarray, x_pre: np.ndarray, treat: np.ndarray) -> dict:
# Estimate theta globally across pooled pre-experiment covariate to keep estimators unbiased under randomization
x_centered =
var_x = float(np.var(x_pre, ddof=1))
cov_yx = float(np.sum((y - np.mean(y)) * x_centered) / max(len(y) - 1, 1))
theta = cov_yx / var_x if var_x > 1e-12 else 0.0
y_cuped = y - theta * x_centered
mask_t = treat == 1
mask_c = treat == 0
ate_raw = float(np.mean(y[mask_t]) - np.mean(y[mask_c]))
ate_cuped = float(np.mean(y_cuped[mask_t]) - np.mean(y_cuped[mask_c]))
se_raw = float(np.sqrt(np.var(y[mask_t], ddof=1) / mask_t.sum() + np.var(y[mask_c], ddof=1) / mask_c.sum()))
se_cuped = float(np.sqrt(np.var(y_cuped[mask_t], ddof=1) / mask_t.sum() + np.var(y_cuped[mask_c], ddof=1) / mask_c.sum()))
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does CUPED connect mathematically to advantage/baseline functions in RL?
Q2: When could CUPED hurt variance?
Next: Vectorized Out-of-Order Event Stream Sessionization (SQL LAG to NumPy) →
💡

Core Mental Anchor / Mnemonic

Pooled Cov/Var removes pre-trend noise unbiasedly by rho-squared

📐 Mathematical Derivation & Core Formula

$$\tilde{Y}_i = Y_i - \theta(X_i - \bar{X}),\quad \theta^* = \frac{\text{Cov}(Y, X)}{\text{Var}(X)},\quad \frac{\text{Var}(\tilde{Y})}{\text{Var}(Y)} = 1 - \rho_{XY}^2$$
Because pre-experiment covariate XX is independent of randomized treatment assignment, subtracting θ(X−Xˉ)\theta(X - \bar{X}) keeps E[Y~T−Y~C]=ATEE[\tilde{Y}_T - \tilde{Y}_C] = \text{ATE} unbiased while shrinking sampling variance by 1−ρXY21 - \rho_{XY}^2 at θ∗=Cov(Y,X)/Var(X)\theta^* = \text{Cov}(Y, X)/\text{Var}(X).

🔄 Tensor Dimensions & Shape Flow

y, x_pre, treat: (N_users,) -> unbiased ATE & reduced SE

🛡️ Numerical Stability & Pitfalls Checklist

  • Use strictly pre-exposure covariates X_pre to prevent post-treatment collider bias
  • Estimate theta on pooled treatment+control samples rather than separately per arm
  • Impute missing X_pre for brand-new users with global mean so sample selection remains unbiased

💻 Reference Code Implementation

import numpy as np

def cuped_ab_evaluation(y: np.ndarray, x_pre: np.ndarray, treat: np.ndarray) -> dict:
    # Estimate theta globally across pooled pre-experiment covariate to keep estimators unbiased under randomization
    x_centered = x_pre - np.mean(x_pre)
    var_x = float(np.var(x_pre, ddof=1))
    cov_yx = float(np.sum((y - np.mean(y)) * x_centered) / max(len(y) - 1, 1))
    theta = cov_yx / var_x if var_x > 1e-12 else 0.0
    y_cuped = y - theta * x_centered
    mask_t = treat == 1
    mask_c = treat == 0
    ate_raw = float(np.mean(y[mask_t]) - np.mean(y[mask_c]))
    ate_cuped = float(np.mean(y_cuped[mask_t]) - np.mean(y_cuped[mask_c]))
    se_raw = float(np.sqrt(np.var(y[mask_t], ddof=1) / mask_t.sum() + np.var(y[mask_c], ddof=1) / mask_c.sum()))
    se_cuped = float(np.sqrt(np.var(y_cuped[mask_t], ddof=1) / mask_t.sum() + np.var(y_cuped[mask_c], ddof=1) / mask_c.sum()))
    return {"theta": theta, "ate_raw": ate_raw, "ate_cuped": ate_cuped, "se_raw": se_raw, "se_cuped": se_cuped, "var_reduction_pct": 100.0 * (1.0 - (se_cuped / max(se_raw, 1e-12)) ** 2)}

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

rng = np.random.default_rng(7)
n = 2000
x = rng.normal(50, 10, n)
treat = rng.integers(0, 2, n)
y = 0.85 * x + 2.0 * treat + rng.normal(0, 3.0, n)
res = cuped_ab_evaluation(y, x, treat)
assert res['var_reduction_pct'] > 70.0
assert res['se_cuped'] < 0.5 * res['se_raw']
print('O3 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does CUPED connect mathematically to advantage/baseline functions in RL?
Both are zero-expectation control variates: subtracting state value V(s) or pre-experiment X removes predictable variance without altering the expected treatment effect or policy gradient.
Q2:When could CUPED hurt variance?
Only when pre-post correlation is near zero and theta estimation noise adds tiny finite-sample overhead; high-traffic experiments easily avoid this or add L2 shrinkage.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Simpson's Paradox Detector & Kitagawa Funnel DecompositionBack to Part O & All 105Next: Vectorized Out-of-Order Event Stream Sessionization (SQL LAG to NumPy) →