M1-053M1: Mathematics & Statistics Fundamentals实验设计 (A/B)Medium
⚡Flashcard Drill➔Mastery:
Define Sample Ratio Mismatch (SRM), How to Detect It via Chi-Square, and Root Cause Diagnostics
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Sample Ratio Mismatch (SRM) occurs when the observed traffic split deviates significantly from the planned design ratio (e.g. 50:50), indicating severe selection bias that invalidates all experimental results.
📌 Key Takeaways
- •Detection: Evaluated using a Chi-Square goodness-of-fit test comparing observed sample sizes to expected ; flagged if .
- •Consequence: Invalidates all causal conclusions; any observed lift could be an artifact of unequal user attrition rather than true product impact.
- •Root Causes: Redirect performance lags, browser cache discrepancies, bot filtering asymmetries, and post-treatment tracking drops.
📐 Mathematical Derivations
Let planned allocation ratio be (e.g. 0.5 : 0.5) with total observed traffic . Expected counts are and . The Chi-Square test statistic is: . For example, if and (a seemingly tiny 0.4% difference), . With 1 degree of freedom, critical value at is ; since , , signaling catastrophic SRM.
🏭 Production Trade-offs
Systematic SRM troubleshooting protocol: (1) Segmented Chi-Square breakdown: Slice by Device (iOS vs Android), Browser (Chrome vs Safari), Country, and User State (New vs Returning). If SRM concentrates in a specific slice (e.g. Android only), inspect platform-specific client code. (2) Redirect vs In-line rendering: Redirect variants suffer from network drops before page load. (3) Bot filtering: If Treatment triggers bot defenses more frequently, bot scrapers are selectively removed from one variant.
⚠️ Common Interview Pitfalls
- ✕Attempting to 're-normalize' or re-weight data to salvage an experiment with severe SRM (unobserved selection bias makes results irrecoverable).
- ✕Using standard threshold for SRM checks (due to huge , use stringent to avoid false platform alarms).
🎯 Interviewer Follow-up Inquiries
- ?Why does a heavy client-side JavaScript bundle in Treatment cause selective drop-off before impression logging?
- ?How does Simpson's paradox manifest when changing allocation percentages mid-experiment?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚