M2-003M2: Classical Machine LearningLinear RegressionMedium
⚡Flashcard Drill➔Mastery:

Explain Multicollinearity, Its Impact on Coefficient Stability and Significance Tests, and Diagnostic Thresholds

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-004 →
📐 Mathematical Definition
Var(w^j)=σ2(1−Rj2)∑(xij−xˉj)2\mathrm{Var}(\hat w_j)=\frac{\sigma^2}{(1-R_j^2)\sum(x_{ij}-\bar x_j)^2}
⚡ One-Sentence Core Answer
Multicollinearity occurs when predictor features are strongly linearly correlated; while it does not harm overall R2R^2 or predictions, it inflates coefficient variances catastrophically, flipping signs and ruining feature interpretability.

📌 Key Takeaways

  • •
    Mechanism: High correlation drives XTXX^T X near-singular (minimum eigenvalue λmin⁡→0\lambda_{\min} \to 0), causing inverse matrix diagonal entries to blow up.
  • •
    Symptoms: Model achieves high overall R2R^2 with significant F-test, yet individual tt-tests fail (p>0.05p > 0.05) with wildly erratic signs.
  • •
    Diagnostics: Variance Inflation Factor VIFj=11−Rj2\text{VIF}_j = \frac{1}{1 - R_j^2}; values VIF>5\text{VIF} > 5 or 1010 indicate severe collinearity.

📐 Mathematical Derivations

Variance Inflation Factor derivation: Let feature xjx_j be regressed on all remaining p−1p-1 features, yielding coefficient of determination Rj2R_j^2. The variance of estimated coefficient β^j\hat{\beta}_j is: Var(β^j)=σ2(n−1)sj2⋅11−Rj2=σ2(n−1)sj2⋅VIFj\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{(n-1)s_j^2} \cdot \frac{1}{1 - R_j^2} = \frac{\sigma^2}{(n-1)s_j^2} \cdot \text{VIF}_j. As xjx_j becomes near-perfectly predictable from other features, Rj2→1R_j^2 \to 1, driving VIFj→∞\text{VIF}_j \to \infty. Consequently, standard error se(β^j)∝VIFj→∞\text{se}(\hat{\beta}_j) \propto \sqrt{\text{VIF}_j} \to \infty. In the tt-statistic tj=β^jse(β^j)t_j = \frac{\hat{\beta}_j}{\text{se}(\hat{\beta}_j)}, the inflated denominator crushes tjt_j toward 0, causing statistically important features to appear completely insignificant.

🏭 Production Trade-offs

Remediation strategies in ML engineering: (1) L2 Regularization (Ridge): Adds λI\lambda I to XTXX^T X, bounding condition numbers and shrinking variance. (2) Feature Pruning: Iteratively drop features with highest VIF or combine redundant variables using PCA. (3) Tree Ensembles (XGBoost / LightGBM): Collinear features do not break trees, but split importance is split arbitrarily between correlated twins, distorting feature attribution (SHAP values share credit).
⚠️ Common Interview Pitfalls
  • ✕
    Concluding a feature is useless simply because its p-value is large in a multicollinear regression.
  • ✕
    Attempting to evaluate individual causal impact βj\beta_j without resolving collinearity.
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Principal Component Regression (PCR) resolve multicollinearity via orthogonal subspace projection?
  • ?
    Why does Ridge regression resolve multicollinearity while L1 Lasso arbitrarily selects one feature and drops the rest?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-002: Enumerate the Four Classical Assumptions of Linear Regression and the Consequences of Their Violation📋 Back to BankNext →M2-004: Explain Why Ridge Regression Mitigates Multicollinearity and Derive Its Closed-Form Solution