M2-069M2: Classical Machine LearningFeature SelectionMedium
⚡Flashcard Drill➔Mastery:

Feature Selection Strategies Under Severe Multicollinearity

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-070 →
📐 Mathematical Definition
VIFj=11−Rj2\text{VIF}_j=\frac{1}{1-R_j^2}
⚡ One-Sentence Core Answer
Diagnose with VIF, correlation heatmaps, or condition indices; resolve via Ridge/ElasticNet regularization, PCA orthogonalization, or hierarchical clustering pruning.

📌 Key Takeaways

  • •
    VIF diagnostic: VIFj=1/(1−Rj2)\text{VIF}_j = 1 / (1 - R_j^2); values >5> 5 or 1010 indicate severe collinearity
  • •
    L1 vs L2: Lasso arbitrarily picks one correlated feature, whereas Ridge shares weights stably
  • •
    Hierarchical clustering: group collinear features by correlation and retain the single most interpretable feature

📐 Mathematical Derivations

Variance Inflation Factor (VIF): For feature xjx_j, regress it against all other features x−jx_{-j}. The variance of the estimated coefficient β^j\hat{\beta}_j is: Var(β^j)=σ2(n−1)sj2⋅11−Rj2=σ2(n−1)sj2⋅VIFj\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{(n-1)s_j^2} \cdot \frac{1}{1 - R_j^2} = \frac{\sigma^2}{(n-1)s_j^2} \cdot \text{VIF}_j. When Rj2→1R_j^2 \to 1, VIFj→∞\text{VIF}_j \to \infty, causing explosive variance in parameter estimates and arbitrary sign flips. Matrix Condition: High condition number κ(XTX)=λmax⁡/λmin⁡≫100\kappa(X^T X) = \lambda_{\max} / \lambda_{\min} \gg 100 indicates near-singularity, making (XTX)−1(X^T X)^{-1} numerically unstable.

🏭 Production Trade-offs

Resolution strategies: ① Regularization: Ridge regression adds λI\lambda I to XTXX^T X, guaranteeing non-singular inversion. ElasticNet combines L1 (sparsity) and L2 (grouping effect), keeping groups of correlated features together. ② Clustering & Pruning: Cluster features using Spearman/Pearson correlation distance (1−∣ρ∣1 - |\rho|) and select the feature with the highest univariate target correlation or lowest missing rate from each cluster. ③ Dimensionality Reduction: Apply PCA to project correlated features onto orthogonal principal axes.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming tree models are completely immune to collinearity; while predictions remain accurate, feature importance becomes split and unreliable
  • ✕
    Blindly using Lasso to interpret feature significance, unaware that it randomly drops all but one of a group of collinear features
🎯 Interviewer Follow-up Inquiries
  • ?
    How does ElasticNet address the limitation of Lasso when dealing with groups of highly correlated features?
  • ?
    Why does multicollinearity destabilize coefficient estimation in linear models without affecting overall prediction accuracy?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-068: Why Can Removing 'Useless Features' Sometimes Degrade Model Performance?📋 Back to BankNext →M2-070: How to Evaluate Feature Selection Effectiveness? Key Considerations and Validation