M2-067M2: Classical Machine LearningFeature SelectionEasy
⚡Flashcard Drill➔Mastery:

Comparing Filter, Wrapper, and Embedded Feature Selection Methods

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-068 →
📐 Mathematical Definition
filter: χ2/MI;wrapper: RFE;embedded: LASSO\text{filter}:\ \chi^2/\text{MI};\quad \text{wrapper}:\ \text{RFE};\quad \text{embedded}:\ \text{LASSO}
⚡ One-Sentence Core Answer
Filter methods use statistical metrics independently of models; wrapper methods optimize feature subsets iteratively via model performance; embedded methods integrate selection into training.

📌 Key Takeaways

  • •
    Filter: fast, model-agnostic, ignores feature interactions and joint redundancy
  • •
    Wrapper: high performance, considers interactions, computationally expensive (RFE, forward search)
  • •
    Embedded: optimal balance, performed during model training (Lasso, tree importance)

📐 Mathematical Derivations

Categorization of Feature Selection: ① Filter Methods: Rank features by individual statistical association with the target yy without training downstream models. Metrics include Pearson correlation, ANOVA FF-test, Mutual Information I(X;Y)=∑p(x,y)log⁡p(x,y)p(x)p(y)I(X; Y) = \sum p(x, y) \log \frac{p(x, y)}{p(x)p(y)}, or Chi-Square χ2\chi^2. Cost: O(d)O(d). ② Wrapper Methods: Treat feature selection as a combinatorial search problem using the learning algorithm as an evaluation oracle. Examples: Recursive Feature Elimination (RFE), Sequential Forward Selection (SFS). Cost: O(d2⋅Ttrain)O(d^2 \cdot T_{\text{train}}). ③ Embedded Methods: Feature selection occurs natively during parameter estimation. Examples: L1 regularization min⁡wL(w)+λ∥w∥1\min_w \mathcal{L}(w) + \lambda \|w\|_1 inducing exact sparsity, and GBDT split importance gain.

🏭 Production Trade-offs

Practical pipeline selection: In high-dimensional regimes (d>10,000d > 10,000), first apply Filter methods to eliminate noise and reduce dimensions to manageable scales (d∼500d \sim 500). Next, apply Embedded regularization or Wrapper RFE to refine the feature subset based on downstream model performance.
⚠️ Common Interview Pitfalls
  • ✕
    Running filter methods or RFE on the entire dataset prior to cross-validation split, introducing data leakage
  • ✕
    Relying solely on univariate filters, which discard features that are informative only in combination (e.g., XOR problem)
🎯 Interviewer Follow-up Inquiries
  • ?
    Why can univariate filter methods fail completely on features that form an XOR relationship with the target?
  • ?
    What is the computational complexity difference between Recursive Feature Elimination (RFE) and Lasso?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-066: What is the Hashing Trick (Feature Hashing)? Pros and Cons📋 Back to BankNext →M2-068: Why Can Removing 'Useless Features' Sometimes Degrade Model Performance?