M1-053M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Medium
⚡Flashcard Drill➔Mastery:

Define Sample Ratio Mismatch (SRM), How to Detect It via Chi-Square, and Root Cause Diagnostics

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M1-054 →
📐 Mathematical Definition
χ2=∑(Oi−Ei)2Ei,p<0.001⇒SRM\chi^2=\sum\frac{(O_i-E_i)^2}{E_i},\qquad p<0.001\Rightarrow\text{SRM}
⚡ One-Sentence Core Answer
Sample Ratio Mismatch (SRM) occurs when the observed traffic split deviates significantly from the planned design ratio (e.g. 50:50), indicating severe selection bias that invalidates all experimental results.

📌 Key Takeaways

  • •
    Detection: Evaluated using a Chi-Square goodness-of-fit test comparing observed sample sizes OA,OBO_A, O_B to expected EA,EBE_A, E_B; flagged if pSRM<0.001p_{\text{SRM}} < 0.001.
  • •
    Consequence: Invalidates all causal conclusions; any observed lift could be an artifact of unequal user attrition rather than true product impact.
  • •
    Root Causes: Redirect performance lags, browser cache discrepancies, bot filtering asymmetries, and post-treatment tracking drops.

📐 Mathematical Derivations

Let planned allocation ratio be rA:rBr_A : r_B (e.g. 0.5 : 0.5) with total observed traffic N=OA+OBN = O_A + O_B. Expected counts are EA=NrAE_A = N r_A and EB=NrBE_B = N r_B. The Chi-Square test statistic is: χ2=(OA−EA)2EA+(OB−EB)2EB=(OA−N/2)2N/2+(OB−N/2)2N/2=(OA−OB)2N∼χ12\chi^2 = \frac{(O_A - E_A)^2}{E_A} + \frac{(O_B - E_B)^2}{E_B} = \frac{(O_A - N/2)^2}{N/2} + \frac{(O_B - N/2)^2}{N/2} = \frac{(O_A - O_B)^2}{N} \sim \chi^2_1. For example, if OA=502,000O_A = 502,000 and OB=498,000O_B = 498,000 (a seemingly tiny 0.4% difference), χ2=(4000)21,000,000=16.0\chi^2 = \frac{(4000)^2}{1,000,000} = 16.0. With 1 degree of freedom, critical value at α=0.001\alpha=0.001 is 10.8310.83; since 16.0>10.8316.0 > 10.83, pSRM=0.000063p_{\text{SRM}} = 0.000063, signaling catastrophic SRM.

🏭 Production Trade-offs

Systematic SRM troubleshooting protocol: (1) Segmented Chi-Square breakdown: Slice by Device (iOS vs Android), Browser (Chrome vs Safari), Country, and User State (New vs Returning). If SRM concentrates in a specific slice (e.g. Android only), inspect platform-specific client code. (2) Redirect vs In-line rendering: Redirect variants suffer from network drops before page load. (3) Bot filtering: If Treatment triggers bot defenses more frequently, bot scrapers are selectively removed from one variant.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to 're-normalize' or re-weight data to salvage an experiment with severe SRM (unobserved selection bias makes results irrecoverable).
  • ✕
    Using standard p=0.05p=0.05 threshold for SRM checks (due to huge NN, use stringent p<0.001p < 0.001 to avoid false platform alarms).
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does a heavy client-side JavaScript bundle in Treatment cause selective drop-off before impression logging?
  • ?
    How does Simpson's paradox manifest when changing allocation percentages mid-experiment?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM1-052: Explain the Purpose and Diagnostic Criteria of A/A Testing in Online Experimentation📋 Back to BankNext →M1-054: Explain CUPED (Controlled-experiment Using Pre-Experiment Data) and Why It Achieves Massive Variance Reduction