M2-091M2: Classical Machine LearningFeature SelectionHard
⚡Flashcard Drill➔Mastery:

Evaluating Feature Importance in Neural Networks vs Tree Models

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-092 →
📐 Mathematical Definition
IG: (x−x′)⋅∫01∇xf(x′+α(x−x′)) dα\text{IG}:\ (x-x')\cdot\int_0^1\nabla_x f(x'+\alpha(x-x'))\,d\alpha
⚡ One-Sentence Core Answer
Neural networks utilize gradient-based attribution (Integrated Gradients, Saliency) and perturbation methods, whereas tree models rely on impurity gain, split count, and TreeSHAP.

📌 Key Takeaways

  • •
    Gradient methods: Integrated Gradients guarantees completeness and implementation invariance
  • •
    Permutation importance: model-agnostic, shuffles features to observe validation score drops
  • •
    TreeSHAP: polynomial-time exact Shapley calculation for trees (O(TLD2)O(T L D^2)) vs path-integral approximation for neural networks

📐 Mathematical Derivations

Methodologies across paradigms:
① Integrated Gradients (IG): Addresses gradient saturation (e.g., in sigmoids or ReLUs) by integrating gradients along the straight-line path from a baseline x′x' to input xx: IGi(x)=(xi−xi′)×∫01∂F(x′+α(x−x′))∂xidα\text{IG}_i(x) = (x_i - x'_i) \times \int_0^1 \frac{\partial F(x' + \alpha(x - x'))}{\partial x_i} d\alpha. In practice, approximated via Riemann sum: IGiapprox(x)=(xi−xi′)×1m∑k=1m∂F(x′+km(x−x′))∂xi\text{IG}_i^{\text{approx}}(x) = (x_i - x'_i) \times \frac{1}{m} \sum_{k=1}^m \frac{\partial F\left(x' + \frac{k}{m}(x - x')\right)}{\partial x_i}. IG satisfies Completeness: ∑iIGi(x)=F(x)−F(x′)\sum_i \text{IG}_i(x) = F(x) - F(x').
② Tree-based Methods: Impurity gain (sum of Gini/variance reduction across splits) and split count. Susceptible to bias favoring high-cardinality features. TreeSHAP solves this by computing exact Shapley values recursively down decision paths.

🏭 Production Trade-offs

Industrial trade-offs: GBDT feature attribution via TreeSHAP is fast and exact. For deep networks, Integrated Gradients requires 50–100 backward passes per sample, imposing significant computational latency during online explanation serving.
⚠️ Common Interview Pitfalls
  • ✕
    Using simple raw gradients ∂y∂xi\frac{\partial y}{\partial x_i} at the input, which suffer from severe gradient saturation in saturated activation regimes
  • ✕
    Choosing an arbitrary or unrepresentative baseline x′x' in Integrated Gradients, which distorts attribution results
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does Integrated Gradients satisfy the Implementation Invariance axiom while simple gradient ×\times input does not?
  • ?
    What computational advantage does TreeSHAP have over KernelSHAP for tree ensembles?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-090: Why Offline and Online Evaluations Diverge: Causes and Mitigation Strategies📋 Back to BankNext →M2-092: Weighted Least Squares (WLS) and Handling Heteroscedasticity