N3Medium★ MLOps & Risk Core · Drift DetectionPart N · Data Pipelines & Production SysOps

Population Stability Index (PSI) Drift Monitor

⚡Flashcard Drill➔

Quantify online vs baseline feature distribution shift using training-derived quantile buckets and clipped symmetric KL divergence.

⏱️ Time: O((N + M) log B)
💾 Space: O(B)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_psi(expected: np.ndarray, actual: np.ndarray, num_bins: int = 10, eps: float = 1e-6) -> float:
"""
Compute Population Stability Index (PSI) between training baseline (expected) and
production distribution (actual) using baseline quantile buckets.
PSI < 0.1: stable; 0.1..0.25: moderate shift; >= 0.25: severe drift alert.
"""
exp = np.asarray(expected, dtype=np.float64)
act = np.asarray(actual, dtype=np.float64)
quantiles =
edges = np.percentile(exp, quantiles)
edges[0], edges[-1] = -np.inf, np.inf
exp_counts, _ = np.histogram(exp, bins=edges)
act_counts, _ = np.histogram(act, bins=edges)
p = np.clip(exp_counts / max(len(exp), 1), eps, None)
q = np.clip(act_counts / max(len(act), 1), eps, None)
p = p / p.sum()
q = q / q.sum()
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why prefer baseline quantile binning over equal-width binning for PSI?
Next: Prefill / Decode Disaggregated Serving Simulator →
💡

Core Mental Anchor / Mnemonic

Freeze edges from baseline quantiles, bound ends with infinities, clip zero bins, alert at >= 0.25.

📐 Mathematical Derivation & Core Formula

PSI=∑b=1B(pb−qb)ln⁡(pbqb)\text{PSI} = \sum_{b=1}^{B} (p_b - q_b) \ln\left(\frac{p_b}{q_b}\right)
### Symmetric KL Feature Drift Monitoring
Because production ground-truth labels lag weeks, PSI tracks input feature shifts immediately using quantile boundaries fit on the training baseline: ∑(qb−pb)ln⁡(qb/pb)\sum (q_b - p_b)\ln(q_b/p_b). Values >0.25> 0.25 signal severe pipeline or population shift.

🔄 Tensor Dimensions & Shape Flow

baseline [N] -> quantile edges [B+1] -> clipped normalized hist (p, q) -> scalar PSI

🛡️ Numerical Stability & Pitfalls Checklist

  • Always anchor bin edges on the baseline training distribution with -inf/+inf outer bounds.
  • Clip zero-count buckets cleanly with eps before taking logarithms.

💻 Reference Code Implementation

import numpy as np

def compute_psi(expected: np.ndarray, actual: np.ndarray, num_bins: int = 10, eps: float = 1e-6) -> float:
    """
    Compute Population Stability Index (PSI) between training baseline (expected) and
    production distribution (actual) using baseline quantile buckets.
    PSI < 0.1: stable; 0.1..0.25: moderate shift; >= 0.25: severe drift alert.
    """
    exp = np.asarray(expected, dtype=np.float64)
    act = np.asarray(actual, dtype=np.float64)
    quantiles = np.linspace(0, 100, num_bins + 1)
    edges = np.percentile(exp, quantiles)
    edges[0], edges[-1] = -np.inf, np.inf
    exp_counts, _ = np.histogram(exp, bins=edges)
    act_counts, _ = np.histogram(act, bins=edges)
    p = np.clip(exp_counts / max(len(exp), 1), eps, None)
    q = np.clip(act_counts / max(len(act), 1), eps, None)
    p = p / p.sum()
    q = q / q.sum()
    return float(np.sum((q - p) * np.log(q / p)))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
rng = np.random.default_rng(7)
base = rng.normal(0.0, 1.0, size=5000)
same_dist = rng.normal(0.0, 1.0, size=5000)
shifted_dist = rng.normal(1.2, 1.0, size=5000)
psi_ok = compute_psi(base, same_dist, num_bins=10)
psi_drift = compute_psi(base, shifted_dist, num_bins=10)
assert psi_ok < 0.05, f"Identical distribution PSI too high: {psi_ok}"
assert psi_drift > 0.25, f"Shifted distribution failed drift alert: {psi_drift}"
print("✓ PSI feature drift detector assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why prefer baseline quantile binning over equal-width binning for PSI?
Equal-width binning leaves sparse outliers populating most bins empty, creating unstable log ratios; quantile binning ensures ~10% expected support per bucket.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Feature Store Point-in-Time AS-OF JoinBack to Part N & All 105Next: Prefill / Decode Disaggregated Serving Simulator →