SRM Detection & Chi-Square Diagnostics

Sample Ratio Mismatch (SRM) Detection & Root-Cause Diagnostics serves as the primary health-check guardrail in online experimentation platforms; SRM occurs when the observed sample counts between control and treatment ($O_T, O_C$) deviate with extreme statistical significance from design traffic ratios (expected $E_T, E_C$, e.g. 50/50); Detection uses Pearson's Chi-Square Goodness-of-Fit test: $\chi^2 = \sum \frac{(O_i - E_i)^2}{E_i}$ ($df = 1$); if $\chi^2 > 3.84$ ($p < 0.001$), the entire experiment is fundamentally invalid and must be terminated immediately; SRM signals critical underlying engineering defects (e.g. redirect tracking loss, client-side crash drops, bot traffic skew, or performance latency timeouts dropping slow connections).

Automated experimentation platform guardrail checks, experimental validity auditing, and debugging engineering instrumentation bugs.

Eliminates catastrophic false discoveries (e.g. a variant appearing to 'win' because a client bug crashed low-end devices, artificially leaving only high-spending users in the bucket).

Derive the simplified Chi-Square statistic $\chi^2 = \frac{(N_T - N_C)^2}{N_T + N_C}$ for symmetric 50/50 split SRM verification?

Detail the 4 root-cause categories of SRM: Tracking Loss, Crash Dropouts, Latency Timeouts, and Bot Skew with diagnostic steps?

Explain how latency-induced dropouts create Survivorship Bias, converting a disastrous UX regression into a phantom metric lift?

Why must user randomization logs trigger precisely at feature exposure rather than generic app launch to prevent dilution and SRM?

Why does industry strictly mandate discarding SRM-compromised experiments rather than attempting post-hoc statistical weighting corrections?