Data Drift & PSI

PSI (Population Stability Index) is the most common metric for monitoring input-distribution drift, quantifying the binned divergence between the baseline (e.g. training) distribution and the current one: $PSI = \sum_i (A_i - B_i)\ln\frac{A_i}{B_i}$, where $A_i$ is the share of actual samples falling in bin $i$ and $B_i$ the expected share; bins are typically deciles of the baseline distribution, empty actual bins are merged with neighbors to avoid division by zero, and PSI is essentially a symmetrized KL divergence. Rule of thumb: $PSI < 0.1$ stable; $0.1 \le PSI < 0.25$ needs investigation; $PSI \ge 0.25$ signals significant drift and triggers retraining. Two drift types: covariate drift ($P(X)$ changed, e.g. a new customer mix) and concept drift ($P(Y|X)$ changed, e.g. default rates rose during COVID).

a must-have on weekly/monthly monitoring dashboards for credit and anti-fraud models; interview questions like "how do you detect model degradation in production" or "what to do when PSI exceeds the threshold"; compute per-feature PSI to rank offending variables, or one overall PSI over the model score, then localize root causes by PSI ranking.

unlike watching only online metrics such as AUC (labels arrive late, and no localization of the offending feature), PSI detects input-distribution drift per feature ahead of labels; combined with tiered thresholds ($<0.1$ stable / $0.1\text{-}0.25$ watch / $>0.25$ drift) and retraining triggers, it closes the monitor—diagnose—retrain loop.

Write $PSI = \sum_i (A_i - B_i)\ln\frac{A_i}{B_i}$ and define $A_i$/$B_i$; why is PSI = 0 when $A_i = B_i$? How does it relate to KL divergence?

PSI thresholds <0.1 / 0.1–0.25 / >0.25 and the action for each; how to handle zero bins ($B_i = 0$ or $A_i = 0$)?

Covariate drift vs concept drift: definitions, one real example each, and how each affects the model differently?

Why monitor PSI instead of online AUC alone? What are PSI's limitations (e.g. sensitivity to binning)?

The action pipeline when PSI ≥ 0.25 (attribute → retrain → rollback); how to distinguish real drift from data leakage or definition changes (feature/口径 change)?