M2-077M2: Classical Machine LearningMissing Values & Data LeakageMedium
⚡Flashcard Drill➔Mastery:

Comparing Mean/Median, KNN, and Multiple Imputation (MICE)

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-078 →
📐 Mathematical Definition
MICE: p(Xmis∣Xobs)=∫p(Xmis∣Xobs,θ)p(θ)dθ\text{MICE}:\ p(X_{mis}\mid X_{obs})=\int p(X_{mis}\mid X_{obs},\theta)p(\theta)d\theta
⚡ One-Sentence Core Answer
Mean/median is fast but underestimates variance; KNN captures local correlations at high inference cost; MICE preserves uncertainty and distribution fidelity through chained equations.

📌 Key Takeaways

  • •
    Mean/median: O(1)O(1) lookup, crushes variance and distorts covariance structures
  • •
    KNN imputation: preserves non-linear neighborhood patterns, O(N)O(N) inference complexity
  • •
    MICE (Multiple Imputation by Chained Equations): models imputation uncertainty by pooling multiple randomized datasets

📐 Mathematical Derivations

Properties across imputation methods: ① Mean/Median Imputation: x^ij=xˉj\hat{x}_{ij} = \bar{x}_j. Limitation: Artificially spikes the distribution mode, deflates sample variance Var(X^)<Var(X)\text{Var}(\hat{X}) < \text{Var}(X), and attenuates Pearson correlation toward zero. ② KNN Imputation: Finds KK nearest neighbors among complete features and computes distance-weighted average: x^ij=∑k∈NK(i)wkxkj\hat{x}_{ij} = \sum_{k \in \mathcal{N}_K(i)} w_k x_{kj}. Captures local manifolds but requires storing training data and computing distances during inference. ③ MICE: Iteratively fits a series of univariate regression models where feature xjx_j is predicted by all other features x−jx_{-j}. Draws MM imputed datasets from posterior predictive distributions: xij(m)∼P(Xj∣X−j,θ(m))x_{ij}^{(m)} \sim P(X_j | X_{-j}, \theta^{(m)}), analyzes each independently, and pools results via Rubin's Rules to reflect parameter and imputation uncertainty.

🏭 Production Trade-offs

Deployment choices: In real-time high-throughput systems, Median + Missing Indicator or native GBDT branch assignment is industry standard due to sub-millisecond SLAs. In offline clinical or econometric studies where parameter confidence intervals are paramount, MICE is mandatory.
⚠️ Common Interview Pitfalls
  • ✕
    Using KNN imputation in production without accounting for the latency of calculating pairwise distances against the entire training dataset
  • ✕
    Fitting imputation statistics on the entire dataset instead of strictly inside the training fold
🎯 Interviewer Follow-up Inquiries
  • ?
    How do Rubin's rules aggregate parameter estimates and variances across multiple imputed datasets in MICE?
  • ?
    Why does mean imputation artificially attenuate correlation coefficients between features?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-076: Three Missing Data Mechanisms (MCAR, MAR, MNAR) and Treatment Strategies📋 Back to BankNext →M2-078: When Should You Add a 'Missing Indicator Variable'?