M2-091M2: Classical Machine LearningFeature SelectionHard
⚡Flashcard Drill➔Mastery:
Evaluating Feature Importance in Neural Networks vs Tree Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Neural networks utilize gradient-based attribution (Integrated Gradients, Saliency) and perturbation methods, whereas tree models rely on impurity gain, split count, and TreeSHAP.
📌 Key Takeaways
- •Gradient methods: Integrated Gradients guarantees completeness and implementation invariance
- •Permutation importance: model-agnostic, shuffles features to observe validation score drops
- •TreeSHAP: polynomial-time exact Shapley calculation for trees () vs path-integral approximation for neural networks
📐 Mathematical Derivations
Methodologies across paradigms:
① Integrated Gradients (IG): Addresses gradient saturation (e.g., in sigmoids or ReLUs) by integrating gradients along the straight-line path from a baseline to input : . In practice, approximated via Riemann sum: . IG satisfies Completeness: .
② Tree-based Methods: Impurity gain (sum of Gini/variance reduction across splits) and split count. Susceptible to bias favoring high-cardinality features. TreeSHAP solves this by computing exact Shapley values recursively down decision paths.
① Integrated Gradients (IG): Addresses gradient saturation (e.g., in sigmoids or ReLUs) by integrating gradients along the straight-line path from a baseline to input : . In practice, approximated via Riemann sum: . IG satisfies Completeness: .
② Tree-based Methods: Impurity gain (sum of Gini/variance reduction across splits) and split count. Susceptible to bias favoring high-cardinality features. TreeSHAP solves this by computing exact Shapley values recursively down decision paths.
🏭 Production Trade-offs
Industrial trade-offs: GBDT feature attribution via TreeSHAP is fast and exact. For deep networks, Integrated Gradients requires 50–100 backward passes per sample, imposing significant computational latency during online explanation serving.
⚠️ Common Interview Pitfalls
- ✕Using simple raw gradients at the input, which suffer from severe gradient saturation in saturated activation regimes
- ✕Choosing an arbitrary or unrepresentative baseline in Integrated Gradients, which distorts attribution results
🎯 Interviewer Follow-up Inquiries
- ?Why does Integrated Gradients satisfy the Implementation Invariance axiom while simple gradient input does not?
- ?What computational advantage does TreeSHAP have over KernelSHAP for tree ensembles?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚