M2-029M2: Classical Machine Learning集成方法 (Bagging/RF)Medium
⚡Flashcard Drill➔Mastery:
Detail Mean Decrease Impurity (MDI) vs. Permutation Importance (MDA) in Random Forests and Their Known Biases
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Feature importance is measured via Mean Decrease Impurity (MDI, summing tree split gains) and Permutation Feature Importance (MDA, measuring score drops after shuffling a column); MDI is severely biased toward high-cardinality features, while MDA is robust.
📌 Key Takeaways
- •Mean Decrease Impurity (MDI / Gini Importance): Sums impurity reductions achieved by feature across all split nodes in all trees.
- •Permutation Feature Importance (MDA / Mean Decrease Accuracy): Shuffles feature values on out-of-bag/test data and measures validation accuracy drop.
- •Flaws of MDI: Severely inflated for continuous or high-cardinality categorical variables; evaluates training fit rather than generalization.
📐 Mathematical Derivations
Mathematical formulation: (1) MDI: , where is node sample fraction. Because high-cardinality features provide many candidate thresholds, they have vastly more opportunities to reduce training impurity by chance, even if pure noise. (2) Permutation Importance (MDA): Let baseline OOB score be . Randomly shuffle column in the OOB set to form , breaking its relationship with target . Evaluate shuffled score . The importance is: . If is critical, shuffling drops score massively; if noise, score drop is .
🏭 Production Trade-offs
Handling Collinear Features: If two features and are strongly collinear (e.g. ), when is shuffled, the model can still access the identical information via unshuffled . Thus, Permutation Importance shows both features as having low importance! In production attribution, SHAP (Shapley Additive Explanations) based on cooperative game theory or grouped permutation importance resolves this collinear masking.
⚠️ Common Interview Pitfalls
- ✕Relying on default
feature_importances_(MDI) in scikit-learn for high-cardinality features (e.g. Random numerical IDs appear at the top of importance rankings). - ✕Computing Permutation Feature Importance on training data instead of test/OOB data.
🎯 Interviewer Follow-up Inquiries
- ?Why does Strobl et al. (2007) prove that MDI is systematically biased toward variables with many categories?
- ?How do TreeSHAP algorithms compute exact Shapley values in polynomial time ?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚