Data Leakage

Data leakage means the training process uses information unavailable at validation/test (or production) time; three types: ① target leakage — features contain the target directly or indirectly (using “already defaulted” to predict default, refund amount to predict refunds); ② temporal leakage — predicting the past with future information (predicting tomorrow's price with today-plus-1 features); ③ pipeline leakage — preprocessing fit on the whole dataset before splitting (scaling, imputation, feature selection, target-encoding statistics have all “seen” the validation folds). Consequences: inflated train and CV scores, abnormal feature importances, and a cliff in production. Three iron rules: scaling/imputation/encoding must be done inside each CV fold (fit on the train fold, transform the validation fold); every derived statistic (means, encodings, selected features) is computed on the train fold only; time-series data is split by time before any processing.

a universal risk in every modeling pipeline; interviews ask “scaling inside or outside CV”, “what does OOF encoding prevent”, “how to detect leakage”; a classic failure point in competitions and industrial risk control.

a “perfect” high-scoring model may have learned no generalizable pattern and fails the moment it ships; encapsulating preprocessing inside the CV pipeline (e.g. sklearn Pipeline fit/transform per fold) keeps validation scores aligned with production — OOF encoding and in-CV scaling exist precisely to close the last invisible leak so offline evaluation is an honest proxy for online performance.

Give a concrete example of each leakage type (target/temporal/pipeline); why do all three inflate CV scores?

Why must scaling happen inside CV (split first, then fit the scaler)? What goes wrong with fit-on-all-then-split?

How do target encoding and group-mean statistics leak? Which leakage does the OOF procedure prevent?

How do you detect leakage (near-zero training error, CV far below training, abnormal importances, AUC near 1)?

What forms can future information take in time-series? How do lag features and rolling statistics leak when misused?