P1Medium★ MLE Production Debugging & Drift DetectionPart P · MLE Production Engineering & Calibration
Training-Serving Skew Audit via Two-Sample KS Test
Implement a SciPy-free empirical two-sample Kolmogorov-Smirnov test with np.searchsorted to catch training-serving distribution skew.
⏱️ Time:
O((N+M) log(N+M))💾 Space:
O(N+M)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def ks_two_sample_skew(train_vals: np.ndarray, serve_vals: np.ndarray) -> dict:
a =
b = np.sort(serve_vals[~np.isnan(serve_vals)])
n, m = len(a), len(b)
if n == 0 or m == 0:
return {"ks_stat": 1.0, "crit_001": 1.0, "skew_detected": True}
grid = np.concatenate([a, b])
cdf_a = np.searchsorted(a, grid, side="right") / n
cdf_b = np.searchsorted(b, grid, side="right") / m
ks_stat = float(np.max(np.abs(cdf_a - cdf_b)))
# alpha=0.01 asymptotic critical value c(alpha)=1.63
crit =
return {"ks_stat": ks_stat, "crit_001": crit, "skew_detected": bool(ks_stat > crit)}
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What usually causes offline test metrics to look great while production AUC craters?
Q2: Why does KS alert on trivial noise when comparing 5 million rows?
💡
Core Mental Anchor / Mnemonic
Sort both sides, right-searchsorted CDF on union grid, take max gap
📐 Mathematical Derivation & Core Formula
$$D_{\text{KS}} = \sup_x |F_{\text{train}}(x) - F_{\text{serve}}(x)|,\quad \text{pass} \iff D_{\text{KS}} < c(\alpha)\sqrt{\frac{n+m}{nm}}$$
The two-sample KS statistic equals the supremum distance between empirical CDFs over all observed points: sorting both non-NaN arrays and querying
np.searchsorted(..., side='right') across the merged grid yields exact in time.🔄 Tensor Dimensions & Shape Flow
train_vals: (N,), serve_vals: (M,) -> right-closed CDF diff -> scalar D_KS
🛡️ Numerical Stability & Pitfalls Checklist
- Pass side='right' to np.searchsorted to represent right-continuous empirical CDFs P(X <= x)
- Combine statistical thresholds with practical effect-size bounds (e.g. D > 0.05) on multi-million-row streams
- Track NaN-rate divergence separately before dropping NaNs
💻 Reference Code Implementation
import numpy as np
def ks_two_sample_skew(train_vals: np.ndarray, serve_vals: np.ndarray) -> dict:
a = np.sort(train_vals[~np.isnan(train_vals)])
b = np.sort(serve_vals[~np.isnan(serve_vals)])
n, m = len(a), len(b)
if n == 0 or m == 0:
return {"ks_stat": 1.0, "crit_001": 1.0, "skew_detected": True}
grid = np.concatenate([a, b])
cdf_a = np.searchsorted(a, grid, side="right") / n
cdf_b = np.searchsorted(b, grid, side="right") / m
ks_stat = float(np.max(np.abs(cdf_a - cdf_b)))
# alpha=0.01 asymptotic critical value c(alpha)=1.63
crit = float(1.63 * np.sqrt((n + m) / (n * m)))
return {"ks_stat": ks_stat, "crit_001": crit, "skew_detected": bool(ks_stat > crit)}🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
rng = np.random.default_rng(1)
tr = rng.normal(0, 1, 1000)
ok = rng.normal(0, 1, 1000)
bad = rng.normal(0.5, 1, 1000)
assert ks_two_sample_skew(tr, ok)['skew_detected'] is False
assert ks_two_sample_skew(tr, bad)['skew_detected'] is True
print('P1 PASS')🎯 Core Follow-up Inquiries & Reference Answers
Q1:What usually causes offline test metrics to look great while production AUC craters?
Training-serving skew: misaligned missing-value defaults, inconsistent float/string tokenization across Python vs Go/C++, or offline leakage from non-point-in-time joins.
Q2:Why does KS alert on trivial noise when comparing 5 million rows?
Standard error shrinks as sqrt(1/n), making negligible 0.5% shifts statistically significant; pair KS with effect-size floors or subsampled fixed windows.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.