P1Medium★ MLE Production Debugging & Drift DetectionPart P · MLE Production Engineering & Calibration

Training-Serving Skew Audit via Two-Sample KS Test

⚡Flashcard Drill➔

Implement a SciPy-free empirical two-sample Kolmogorov-Smirnov test with np.searchsorted to catch training-serving distribution skew.

⏱️ Time: O((N+M) log(N+M))
💾 Space: O(N+M)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def ks_two_sample_skew(train_vals: np.ndarray, serve_vals: np.ndarray) -> dict:
a =
b = np.sort(serve_vals[~np.isnan(serve_vals)])
n, m = len(a), len(b)
if n == 0 or m == 0:
return {"ks_stat": 1.0, "crit_001": 1.0, "skew_detected": True}
grid = np.concatenate([a, b])
cdf_a = np.searchsorted(a, grid, side="right") / n
cdf_b = np.searchsorted(b, grid, side="right") / m
ks_stat = float(np.max(np.abs(cdf_a - cdf_b)))
# alpha=0.01 asymptotic critical value c(alpha)=1.63
crit =
return {"ks_stat": ks_stat, "crit_001": crit, "skew_detected": bool(ks_stat > crit)}
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What usually causes offline test metrics to look great while production AUC craters?
Q2: Why does KS alert on trivial noise when comparing 5 million rows?
Next: Dual-Budget Token & Item Batch Packer for Async Embeddings →
💡

Core Mental Anchor / Mnemonic

Sort both sides, right-searchsorted CDF on union grid, take max gap

📐 Mathematical Derivation & Core Formula

$$D_{\text{KS}} = \sup_x |F_{\text{train}}(x) - F_{\text{serve}}(x)|,\quad \text{pass} \iff D_{\text{KS}} < c(\alpha)\sqrt{\frac{n+m}{nm}}$$
The two-sample KS statistic equals the supremum distance between empirical CDFs over all observed points: sorting both non-NaN arrays and querying np.searchsorted(..., side='right') across the merged grid yields exact Dn,mD_{n,m} in O((n+m)log⁡n)O((n+m)\log n) time.

🔄 Tensor Dimensions & Shape Flow

train_vals: (N,), serve_vals: (M,) -> right-closed CDF diff -> scalar D_KS

🛡️ Numerical Stability & Pitfalls Checklist

  • Pass side='right' to np.searchsorted to represent right-continuous empirical CDFs P(X <= x)
  • Combine statistical thresholds with practical effect-size bounds (e.g. D > 0.05) on multi-million-row streams
  • Track NaN-rate divergence separately before dropping NaNs

💻 Reference Code Implementation

import numpy as np

def ks_two_sample_skew(train_vals: np.ndarray, serve_vals: np.ndarray) -> dict:
    a = np.sort(train_vals[~np.isnan(train_vals)])
    b = np.sort(serve_vals[~np.isnan(serve_vals)])
    n, m = len(a), len(b)
    if n == 0 or m == 0:
        return {"ks_stat": 1.0, "crit_001": 1.0, "skew_detected": True}
    grid = np.concatenate([a, b])
    cdf_a = np.searchsorted(a, grid, side="right") / n
    cdf_b = np.searchsorted(b, grid, side="right") / m
    ks_stat = float(np.max(np.abs(cdf_a - cdf_b)))
    # alpha=0.01 asymptotic critical value c(alpha)=1.63
    crit = float(1.63 * np.sqrt((n + m) / (n * m)))
    return {"ks_stat": ks_stat, "crit_001": crit, "skew_detected": bool(ks_stat > crit)}

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

rng = np.random.default_rng(1)
tr = rng.normal(0, 1, 1000)
ok = rng.normal(0, 1, 1000)
bad = rng.normal(0.5, 1, 1000)
assert ks_two_sample_skew(tr, ok)['skew_detected'] is False
assert ks_two_sample_skew(tr, bad)['skew_detected'] is True
print('P1 PASS')

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What usually causes offline test metrics to look great while production AUC craters?
Training-serving skew: misaligned missing-value defaults, inconsistent float/string tokenization across Python vs Go/C++, or offline leakage from non-point-in-time joins.
Q2:Why does KS alert on trivial noise when comparing 5 million rows?
Standard error shrinks as sqrt(1/n), making negligible 0.5% shifts statistically significant; pair KS with effect-size floors or subsampled fixed windows.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Robust Data Contract Gate (NaN Rate + Median-MAD Z-Score)Back to Part P & All 105Next: Dual-Budget Token & Item Batch Packer for Async Embeddings →