M2-079M2: Classical Machine LearningMissing Values & Data LeakageHard
⚡Flashcard Drill➔Mastery:

How to Systematically Audit Data Leakage? An Actionable Checklist

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-080 →
📐 Mathematical Definition
random label test: 若仍高 AUC⇒泄漏\text{random label test}:\ \text{若仍高 AUC}\Rightarrow\text{泄漏}
⚡ One-Sentence Core Answer
Audit temporal splits, preprocessing pipelines, target proxies, duplicate entities, and ID features to prevent future or test information from leaking into training.

📌 Key Takeaways

  • •
    Temporal leakage: ensuring all features exist strictly prior to the prediction timestamp
  • •
    Pipeline leakage: encapsulating all transformations (scaling, imputation, target encoding) inside CV folds
  • •
    Target proxy leakage: detecting features that are consequences rather than causes of the target label

📐 Mathematical Derivations

Mathematical definition of leakage: Let StrainS_{\text{train}} and StestS_{\text{test}} be sample spaces. Leakage occurs when the feature distribution in training is conditioned on information not available at test time: P(Xtrain∣Ytest)≠P(Xtrain)P(X_{\text{train}} | Y_{\text{test}}) \ne P(X_{\text{train}}) or when XX contains a direct surrogate ZZ such that I(Z;Y)≈1I(Z; Y) \approx 1 where ZZ is created concurrently with or after YY.
Actionable Audit Checklist:
1. Split Isolation: Did train/val/test split happen BEFORE any normalization, imputation, PCA, or target encoding?
2. Time Integrity: For time-series data, is tfeature<tevent−tlatencyt_{\text{feature}} < t_{\text{event}} - t_{\text{latency}} strictly enforced?
3. Entity Overlap: Are patient/user IDs grouped (GroupKFold) so the same entity never appears in both train and validation?
4. Unrealistic Feature Importance: Does a single feature exhibit >0.95>0.95 correlation or dominate tree importance (often a target proxy, e.g., 'refund_date' predicting 'cancellation')?
5. Metric Discrepancy: Is cross-validation performance suspiciously higher than historical baselines or sudden drop on out-of-time evaluation?

🏭 Production Trade-offs

Standardized defense: Wrap all feature preprocessing inside sklearn.pipeline.Pipeline. For temporal data, strictly forbid randomized K-fold; enforce TimeSeriesSplit. Implement automated pre-merge unit tests that check correlation between features and the target label.
⚠️ Common Interview Pitfalls
  • ✕
    Normalizing or vectorizing text across the entire dataset before creating train/validation splits
  • ✕
    Including post-event attributes (such as account deactivation timestamp) as features when predicting customer churn
🎯 Interviewer Follow-up Inquiries
  • ?
    How does GroupKFold protect against data leakage when multiple rows belong to the same user or patient?
  • ?
    What automated tests can be added to CI/CD pipelines to catch target leakage before production deployment?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-078: When Should You Add a 'Missing Indicator Variable'?📋 Back to BankNext →M2-080: Why Global Normalization Before Splitting Causes Data Leakage and How to Fix It