N3Medium★ MLOps & Risk Core · Drift DetectionPart N · Data Pipelines & Production SysOps
Population Stability Index (PSI) Drift Monitor
Quantify online vs baseline feature distribution shift using training-derived quantile buckets and clipped symmetric KL divergence.
⏱️ Time:
O((N + M) log B)💾 Space:
O(B)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_psi(expected: np.ndarray, actual: np.ndarray, num_bins: int = 10, eps: float = 1e-6) -> float:
"""
Compute Population Stability Index (PSI) between training baseline (expected) and
production distribution (actual) using baseline quantile buckets.
PSI < 0.1: stable; 0.1..0.25: moderate shift; >= 0.25: severe drift alert.
"""
exp = np.asarray(expected, dtype=np.float64)
act = np.asarray(actual, dtype=np.float64)
quantiles =
edges = np.percentile(exp, quantiles)
edges[0], edges[-1] = -np.inf, np.inf
exp_counts, _ = np.histogram(exp, bins=edges)
act_counts, _ = np.histogram(act, bins=edges)
p = np.clip(exp_counts / max(len(exp), 1), eps, None)
q = np.clip(act_counts / max(len(act), 1), eps, None)
p = p / p.sum()
q = q / q.sum()
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why prefer baseline quantile binning over equal-width binning for PSI?
💡
Core Mental Anchor / Mnemonic
Freeze edges from baseline quantiles, bound ends with infinities, clip zero bins, alert at >= 0.25.
📐 Mathematical Derivation & Core Formula
### Symmetric KL Feature Drift Monitoring
Because production ground-truth labels lag weeks, PSI tracks input feature shifts immediately using quantile boundaries fit on the training baseline: . Values signal severe pipeline or population shift.
Because production ground-truth labels lag weeks, PSI tracks input feature shifts immediately using quantile boundaries fit on the training baseline: . Values signal severe pipeline or population shift.
🔄 Tensor Dimensions & Shape Flow
baseline [N] -> quantile edges [B+1] -> clipped normalized hist (p, q) -> scalar PSI
🛡️ Numerical Stability & Pitfalls Checklist
- Always anchor bin edges on the baseline training distribution with
-inf/+infouter bounds. - Clip zero-count buckets cleanly with
epsbefore taking logarithms.
💻 Reference Code Implementation
import numpy as np
def compute_psi(expected: np.ndarray, actual: np.ndarray, num_bins: int = 10, eps: float = 1e-6) -> float:
"""
Compute Population Stability Index (PSI) between training baseline (expected) and
production distribution (actual) using baseline quantile buckets.
PSI < 0.1: stable; 0.1..0.25: moderate shift; >= 0.25: severe drift alert.
"""
exp = np.asarray(expected, dtype=np.float64)
act = np.asarray(actual, dtype=np.float64)
quantiles = np.linspace(0, 100, num_bins + 1)
edges = np.percentile(exp, quantiles)
edges[0], edges[-1] = -np.inf, np.inf
exp_counts, _ = np.histogram(exp, bins=edges)
act_counts, _ = np.histogram(act, bins=edges)
p = np.clip(exp_counts / max(len(exp), 1), eps, None)
q = np.clip(act_counts / max(len(act), 1), eps, None)
p = p / p.sum()
q = q / q.sum()
return float(np.sum((q - p) * np.log(q / p)))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
rng = np.random.default_rng(7)
base = rng.normal(0.0, 1.0, size=5000)
same_dist = rng.normal(0.0, 1.0, size=5000)
shifted_dist = rng.normal(1.2, 1.0, size=5000)
psi_ok = compute_psi(base, same_dist, num_bins=10)
psi_drift = compute_psi(base, shifted_dist, num_bins=10)
assert psi_ok < 0.05, f"Identical distribution PSI too high: {psi_ok}"
assert psi_drift > 0.25, f"Shifted distribution failed drift alert: {psi_drift}"
print("✓ PSI feature drift detector assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why prefer baseline quantile binning over equal-width binning for PSI?
Equal-width binning leaves sparse outliers populating most bins empty, creating unstable log ratios; quantile binning ensures ~10% expected support per bucket.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.