📊 Classic MLID: vif-multicollinearity

VIF Multicollinearity

Q: Derive VIF: why $VIF_j = \frac{1}{1-R_j^2}$? Where does $R_j^2$ come from, and what $R_j^2$ does the threshold 10 correspond to?

【VIF Multicollinearity 核心考点 1】Derive VIF: why $VIF_j = \frac{1}{1-R_j^2}$? Where does $R_j^2$ come from, and what $R_j^2$ does the threshold 10 correspond to?。技术解析：Definition: multicollinearity means features are nearly linearly dependent, making X^TX close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress x_j on the other p-1 features to obtain R_j^2, then VIF_j = \frac{1}{1 - R_j^2}. As R_j^2 \to 1, VIF blo... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/ML/card/vif-multicollinearity

Q: Derive $Var(\hat{w}_j) = \frac{\sigma^2}{(1-R_j^2)\sum_i (x_{ij}-\bar{x}_j)^2}$ and explain how collinearity inflates variance.

【VIF Multicollinearity 核心考点 2】Derive $Var(\hat{w}_j) = \frac{\sigma^2}{(1-R_j^2)\sum_i (x_{ij}-\bar{x}_j)^2}$ and explain how collinearity inflates variance.。技术解析：Definition: multicollinearity means features are nearly linearly dependent, making X^TX close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress x_j on the other p-1 features to obtain R_j^2, then VIF_j = \frac{1}{1 - R_j^2}. As R_j^2 \to 1, VIF blo... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/ML/card/vif-multicollinearity

Q: Why do predictions stay good under multicollinearity? What exactly does it hurt (coefficients, standard errors, t-tests, feature importance)?

【VIF Multicollinearity 核心考点 3】Why do predictions stay good under multicollinearity? What exactly does it hurt (coefficients, standard errors, t-tests, feature importance)?。技术解析：Definition: multicollinearity means features are nearly linearly dependent, making X^TX close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress x_j on the other p-1 features to obtain R_j^2, then VIF_j = \frac{1}{1 - R_j^2}. As R_j^2 \to 1, VIF blo... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/ML/card/vif-multicollinearity

Q: Compare remedies for collinearity: dropping features vs PCA vs ridge regression — cost and suitability of each.

【VIF Multicollinearity 核心考点 4】Compare remedies for collinearity: dropping features vs PCA vs ridge regression — cost and suitability of each.。技术解析：Definition: multicollinearity means features are nearly linearly dependent, making X^TX close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress x_j on the other p-1 features to obtain R_j^2, then VIF_j = \frac{1}{1 - R_j^2}. As R_j^2 \to 1, VIF blo... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/ML/card/vif-multicollinearity

Q: Relationship between VIF, pairwise feature correlation and the condition number $\kappa(X) = \frac{\lambda_{max}}{\lambda_{min}}$; why are tree models immune?

【VIF Multicollinearity 核心考点 5】Relationship between VIF, pairwise feature correlation and the condition number $\kappa(X) = \frac{\lambda_{max}}{\lambda_{min}}$; why are tree models immune?。技术解析：Definition: multicollinearity means features are nearly linearly dependent, making X^TX close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress x_j on the other p-1 features to obtain R_j^2, then VIF_j = \frac{1}{1 - R_j^2}. As R_j^2 \to 1, VIF blo... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/ML/card/vif-multicollinearity

VIF 多重共线性

🎯Core Definition

multicollinearity means features are nearly linearly dependent, making

X^TX

close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress

x_j

on the other

p-1

features to obtain

R_j^2

, then

VIF_j = \frac{1}{1 - R_j^2}

. As

R_j^2 \to 1

, VIF blows up; rule of thumb:

VIF_j > 10

(i.e.

R_j^2 > 0.9

) is severe,

5 \sim 10

moderate.

💡Use Cases

a routine pre-modeling diagnostic, especially for interpretable models (tree models are naturally immune); follow-up questions include deriving the link between

VIF_j

and coefficient variance, distinguishing VIF from pairwise correlation / condition number, and why predictions can still be fine under collinearity.

⚡Key Problems Solved

collinearity inflates coefficient variance by exactly

VIF_j

— in OLS

Var(\hat{w}_j) = \frac{\sigma^2}{(1 - R_j^2)\sum_i (x_{ij} - \bar{x}_j)^2}

, so a small

1-R_j^2

explodes the variance, producing sign flips, huge standard errors and broken t-tests, while

R^2

and predictions barely change — a prediction-only view misses it. Fixes: drop high-VIF features, PCA / factor analysis, ridge regression (adding

\lambda I

forces full rank), or collect more orthogonal data.

🎯5 High-Frequency Exam Points

Derive VIF: why

VIF_j = \frac{1}{1-R_j^2}

? Where does

R_j^2

come from, and what

R_j^2

does the threshold 10 correspond to?

Derive

Var(\hat{w}_j) = \frac{\sigma^2}{(1-R_j^2)\sum_i (x_{ij}-\bar{x}_j)^2}

and explain how collinearity inflates variance.

Why do predictions stay good under multicollinearity? What exactly does it hurt (coefficients, standard errors, t-tests, feature importance)?

Compare remedies for collinearity: dropping features vs PCA vs ridge regression — cost and suitability of each.

Relationship between VIF, pairwise feature correlation and the condition number

\kappa(X) = \frac{\lambda_{max}}{\lambda_{min}}

; why are tree models immune?

📖 In-depth Guide:📄 linear-and-logistic-regression →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "VIF Multicollinearity"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardLinear Regression OLS Next Card →Logistic Regression

🔗 More Classic ML Knowledge Cards

AdaBoost Derivation Bagging & Random Forest Baum-Welch (HMM EM)GBDT Negative Gradient