SRM Detection & Chi-Square Diagnostics

Sample Ratio Mismatch (SRM) Detection & Root-Cause Diagnostics serves as the primary health-check guardrail in online experimentation platforms; SRM occurs when the observed sample counts between control and treatment ($O_T, O_C$) deviate with extreme statistical significance from design traffic ratios (expected $E_T, E_C$, e.g. 50/50); Detection uses Pearson's Chi-Square Goodness-of-Fit test: $\chi^2 = \sum \frac{(O_i - E_i)^2}{E_i}$ ($df = 1$); if $\chi^2 > 3.84$ ($p < 0.001$), the entire experiment is fundamentally invalid and must be terminated immediately; SRM signals critical underlying engineering defects (e.g. redirect tracking loss, client-side crash drops, bot traffic skew, or performance latency timeouts dropping slow connections).

Automated experimentation platform guardrail checks, experimental validity auditing, and debugging engineering instrumentation bugs.

Eliminates catastrophic false discoveries (e.g. a variant appearing to 'win' because a client bug crashed low-end devices, artificially leaving only high-spending users in the bucket).

Test Your Knowledge: Practice Questions for "SRM Detection & Chi-Square Diagnostics"

Single choice pitfall questions with instant feedback and mistake tracking.