🏗️ System DesignID:
recsys-ab-testing-interleavingOnline A/B Testing & Interleaving
推荐在线 A/B 实验与 Interleaving
🎯Core Definition
Online A/B Testing & Interleaving constitute the gold-standard causal decision frameworks evaluating if algorithmic revisions deliver authentic business impact before 100% rollout; 1) Standard A/B Testing: hashes user traffic into Control and Treatment buckets, running double-blind for days to track primary metrics (CTR, GMV, Dwell Time, Retention) and Guardrail metrics (crash rate, latency), evaluating statistical significance via two-sample t-tests (p-value < 0.05); 2) Interleaving (e.g., Team Draft Interleaving): merges rankings from Model A and Model B into a single interleaved feed presented to the same user, measuring immediate relative clicks with 100x higher sensitivity than standard A/B.
💡Use Cases
Pre-launch canary ramp-up, rapid candidate model culling, and rigorous causal measurement of revenue and engagement KPIs.
⚡Key Problems Solved
Offline metric improvements often fail to translate to online business gains; Interleaving culls weak models within hours using tiny traffic fractions, while A/B testing certifies long-term causal business lift.
🎯5 High-Frequency Exam Points
1
Detail the Team Draft Interleaving algorithm and unbiased credit assignment for user clicks?
2
Why does Interleaving achieve identical statistical power with 1% sample size and 1/10 the duration of standard A/B tests?
3
Explain the architecture of multi-layer orthogonal experiment frameworks (Google's layered hashing) for running concurrent tests?
4
How to detect Sample Ratio Mismatch (SRM) via Chi-Square tests and mitigate network spillover in social networks?
5
How to isolate the Novelty Effect and measure the Slow Cooker Effect over multi-week experimentation windows?
📖 In-depth Guide:📄 recommendation-system-design →
Updated 2026-08-14🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "Online A/B Testing & Interleaving"
Single choice pitfall questions with instant feedback and mistake tracking.