M2-029M2: Classical Machine Learning集成方法 (Bagging/RF)Medium
⚡Flashcard Drill➔Mastery:

Detail Mean Decrease Impurity (MDI) vs. Permutation Importance (MDA) in Random Forests and Their Known Biases

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-030 →
📐 Mathematical Definition
MDIj=∑node split on jΔimpurity\text{MDI}_j=\sum_{\text{node split on }j}\Delta\text{impurity}
⚡ One-Sentence Core Answer
Feature importance is measured via Mean Decrease Impurity (MDI, summing tree split gains) and Permutation Feature Importance (MDA, measuring score drops after shuffling a column); MDI is severely biased toward high-cardinality features, while MDA is robust.

📌 Key Takeaways

  • •
    Mean Decrease Impurity (MDI / Gini Importance): Sums impurity reductions ΔI\Delta I achieved by feature jj across all split nodes in all trees.
  • •
    Permutation Feature Importance (MDA / Mean Decrease Accuracy): Shuffles feature jj values on out-of-bag/test data and measures validation accuracy drop.
  • •
    Flaws of MDI: Severely inflated for continuous or high-cardinality categorical variables; evaluates training fit rather than generalization.

📐 Mathematical Derivations

Mathematical formulation: (1) MDI: MDI(xj)=1B∑b=1B∑t∈Tb:v(t)=jp(t)ΔI(t)\text{MDI}(x_j) = \frac{1}{B}\sum_{b=1}^B \sum_{t \in T_b : v(t)=j} p(t) \Delta I(t), where p(t)=Nt/Np(t) = N_t / N is node sample fraction. Because high-cardinality features provide many candidate thresholds, they have vastly more opportunities to reduce training impurity by chance, even if pure noise. (2) Permutation Importance (MDA): Let baseline OOB score be SOOBS_{\text{OOB}}. Randomly shuffle column jj in the OOB set to form X~(j)\tilde{X}^{(j)}, breaking its relationship with target yy. Evaluate shuffled score S~OOB(j)\tilde{S}_{\text{OOB}}^{(j)}. The importance is: MDA(xj)=SOOB−S~OOB(j)\text{MDA}(x_j) = S_{\text{OOB}} - \tilde{S}_{\text{OOB}}^{(j)}. If xjx_j is critical, shuffling drops score massively; if noise, score drop is ≈0\approx 0.

🏭 Production Trade-offs

Handling Collinear Features: If two features x1x_1 and x2x_2 are strongly collinear (e.g. ρ=0.99\rho = 0.99), when x1x_1 is shuffled, the model can still access the identical information via unshuffled x2x_2. Thus, Permutation Importance shows both features as having low importance! In production attribution, SHAP (Shapley Additive Explanations) based on cooperative game theory or grouped permutation importance resolves this collinear masking.
⚠️ Common Interview Pitfalls
  • ✕
    Relying on default feature_importances_ (MDI) in scikit-learn for high-cardinality features (e.g. Random numerical IDs appear at the top of importance rankings).
  • ✕
    Computing Permutation Feature Importance on training data instead of test/OOB data.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Strobl et al. (2007) prove that MDI is systematically biased toward variables with many categories?
  • ?
    How do TreeSHAP algorithms compute exact Shapley values in polynomial time O(TLD2)O(T L D^2)?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-028: Explain the Two Sources of Randomness in Random Forests and How They Decorrelate Trees📋 Back to BankNext →M2-030: Explain Out-of-Bag (OOB) Error Estimation and Its Practical Utility in Random Forests