M2-079M2: Classical Machine LearningMissing Values & Data LeakageHard
⚡Flashcard Drill➔Mastery:
How to Systematically Audit Data Leakage? An Actionable Checklist
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Audit temporal splits, preprocessing pipelines, target proxies, duplicate entities, and ID features to prevent future or test information from leaking into training.
📌 Key Takeaways
- •Temporal leakage: ensuring all features exist strictly prior to the prediction timestamp
- •Pipeline leakage: encapsulating all transformations (scaling, imputation, target encoding) inside CV folds
- •Target proxy leakage: detecting features that are consequences rather than causes of the target label
📐 Mathematical Derivations
Mathematical definition of leakage: Let and be sample spaces. Leakage occurs when the feature distribution in training is conditioned on information not available at test time: or when contains a direct surrogate such that where is created concurrently with or after .
Actionable Audit Checklist:
1. Split Isolation: Did train/val/test split happen BEFORE any normalization, imputation, PCA, or target encoding?
2. Time Integrity: For time-series data, is strictly enforced?
3. Entity Overlap: Are patient/user IDs grouped (GroupKFold) so the same entity never appears in both train and validation?
4. Unrealistic Feature Importance: Does a single feature exhibit correlation or dominate tree importance (often a target proxy, e.g., 'refund_date' predicting 'cancellation')?
5. Metric Discrepancy: Is cross-validation performance suspiciously higher than historical baselines or sudden drop on out-of-time evaluation?
Actionable Audit Checklist:
1. Split Isolation: Did train/val/test split happen BEFORE any normalization, imputation, PCA, or target encoding?
2. Time Integrity: For time-series data, is strictly enforced?
3. Entity Overlap: Are patient/user IDs grouped (GroupKFold) so the same entity never appears in both train and validation?
4. Unrealistic Feature Importance: Does a single feature exhibit correlation or dominate tree importance (often a target proxy, e.g., 'refund_date' predicting 'cancellation')?
5. Metric Discrepancy: Is cross-validation performance suspiciously higher than historical baselines or sudden drop on out-of-time evaluation?
🏭 Production Trade-offs
Standardized defense: Wrap all feature preprocessing inside
sklearn.pipeline.Pipeline. For temporal data, strictly forbid randomized K-fold; enforce TimeSeriesSplit. Implement automated pre-merge unit tests that check correlation between features and the target label.⚠️ Common Interview Pitfalls
- ✕Normalizing or vectorizing text across the entire dataset before creating train/validation splits
- ✕Including post-event attributes (such as account deactivation timestamp) as features when predicting customer churn
🎯 Interviewer Follow-up Inquiries
- ?How does GroupKFold protect against data leakage when multiple rows belong to the same user or patient?
- ?What automated tests can be added to CI/CD pipelines to catch target leakage before production deployment?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚