VIF Multicollinearity

multicollinearity means features are nearly linearly dependent, making $X^TX$ close to singular. VIF (Variance Inflation Factor) diagnoses it quantitatively: regress $x_j$ on the other $p-1$ features to obtain $R_j^2$, then $VIF_j = \frac{1}{1 - R_j^2}$. As $R_j^2 \to 1$, VIF blows up; rule of thumb: $VIF_j > 10$ (i.e. $R_j^2 > 0.9$) is severe, $5 \sim 10$ moderate.

a routine pre-modeling diagnostic, especially for interpretable models (tree models are naturally immune); follow-up questions include deriving the link between $VIF_j$ and coefficient variance, distinguishing VIF from pairwise correlation / condition number, and why predictions can still be fine under collinearity.

collinearity inflates coefficient variance by exactly $VIF_j$ — in OLS $Var(\hat{w}_j) = \frac{\sigma^2}{(1 - R_j^2)\sum_i (x_{ij} - \bar{x}_j)^2}$, so a small $1-R_j^2$ explodes the variance, producing sign flips, huge standard errors and broken t-tests, while $R^2$ and predictions barely change — a prediction-only view misses it. Fixes: drop high-VIF features, PCA / factor analysis, ridge regression (adding $\lambda I$ forces full rank), or collect more orthogonal data.

Derive VIF: why $VIF_j = \frac{1}{1-R_j^2}$? Where does $R_j^2$ come from, and what $R_j^2$ does the threshold 10 correspond to?

Derive $Var(\hat{w}_j) = \frac{\sigma^2}{(1-R_j^2)\sum_i (x_{ij}-\bar{x}_j)^2}$ and explain how collinearity inflates variance.

Why do predictions stay good under multicollinearity? What exactly does it hurt (coefficients, standard errors, t-tests, feature importance)?

Compare remedies for collinearity: dropping features vs PCA vs ridge regression — cost and suitability of each.

Relationship between VIF, pairwise feature correlation and the condition number $\kappa(X) = \frac{\lambda_{max}}{\lambda_{min}}$; why are tree models immune?