Data Leakage

Data leakage means the training process uses information unavailable at validation/test (or production) time; three types: ① target leakage — features contain the target directly or indirectly (using “already defaulted” to predict default, refund amount to predict refunds); ② temporal leakage — predicting the past with future information (predicting tomorrow's price with today-plus-1 features); ③ pipeline leakage — preprocessing fit on the whole dataset before splitting (scaling, imputation, feature selection, target-encoding statistics have all “seen” the validation folds). Consequences: inflated train and CV scores, abnormal feature importances, and a cliff in production. Three iron rules: scaling/imputation/encoding must be done inside each CV fold (fit on the train fold, transform the validation fold); every derived statistic (means, encodings, selected features) is computed on the train fold only; time-series data is split by time before any processing.

a universal risk in every modeling pipeline; interviews ask “scaling inside or outside CV”, “what does OOF encoding prevent”, “how to detect leakage”; a classic failure point in competitions and industrial risk control.

a “perfect” high-scoring model may have learned no generalizable pattern and fails the moment it ships; encapsulating preprocessing inside the CV pipeline (e.g. sklearn Pipeline fit/transform per fold) keeps validation scores aligned with production — OOF encoding and in-CV scaling exist precisely to close the last invisible leak so offline evaluation is an honest proxy for online performance.

Test Your Knowledge: Practice Questions for "Data Leakage"

Single choice pitfall questions with instant feedback and mistake tracking.