Feature Selection, Permutation & SHAP

Feature Selection Taxonomy, Permutation Importance & SHAP Values provide the systematic framework to compress high-dimensional feature spaces, purge collinear noise, and attribute individual feature contributions via game theory; the 3 selection paradigms comprise: 1) Filter Methods (Mutual Information, Chi-Square, Pearson Correlation); 2) Wrapper Methods (Recursive Feature Elimination, Forward/Backward search); 3) Embedded Methods (L1 Lasso sparsity, Tree Gini Gain); Permutation Importance shuffles a single feature column on validation sets to quantify true performance drops; SHAP (SHapley Additive exPlanations) computes exact Shapley values $\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F|-|S|-1)!}{|F|!} [f(S \cup \{i\}) - f(S)]$, delivering game-theoretically consistent feature attribution.

High-dimensional feature pruning, credit risk regulatory compliance explanations, and diagnosing multicollinearity.

Default tree feature importance (Split Counts) is heavily biased towards high-cardinality continuous variables; Permutation and TreeSHAP yield unbiased, mathematically proven global and local feature attributions.

Derive Shapley value axiomatic properties (Efficiency, Symmetry, Dummy, Additivity) and explain its marginal contribution formulation?

Explain why GBDT Split Count importance is biased towards continuous features with many unique values, and how TreeSHAP corrects this?

Explain how multi-collinearity distorts Permutation Importance and how Hierarchical Clustering Ward linkage groups redundant features?

How does TreeSHAP optimize exponential Shapley calculation down to polynomial time $O(T L D^2)$ using decision tree path structures?

Explain the algorithmic workflow of RFECV (Recursive Feature Elimination with Cross-Validation) in selecting optimal feature subsets?