M2-031M2: Classical Machine Learning集成方法 (Bagging/RF)Hard
⚡Flashcard Drill➔Mastery:
Define Stacking vs. Blending in Heterogeneous Ensembles and Compare Their Information Leakage Safeguards
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Stacking trains a meta-learner on out-of-fold cross-validation predictions from multiple diverse base models, while Blending fits the meta-learner on a static holdout set.
📌 Key Takeaways
- •Stacking (Wolpert, 1992): Generates out-of-fold (OOF) feature matrix across folds; trains meta-model on all OOF predictions.
- •Blending: Splits data into train and holdout validation sets; base models predict on holdout, which forms the training set for the meta-learner.
- •Comparison: Stacking utilizes 100% of data efficiently without waste; Blending is simpler and faster but discards valuable training samples.
📐 Mathematical Derivations
Stacking protocol: Given base models and -fold partition . For each fold : train base models on and generate out-of-fold predictions on : for . Concatenating across all folds constructs the meta-feature matrix . The meta-learner (e.g. Ridge Regression or shallow GBDT) minimizes . At test time, base models trained on full predict on , which are fed into .
🏭 Production Trade-offs
Engineering best practices: (1) Model Diversity: Stacking works best when base models have orthogonal inductive biases (e.g. Blending LightGBM + CatBoost + Neural Network + Logistic Regression). Combining 10 variants of the same XGBoost tree yields near-zero stacking lift. (2) Meta-learner simplicity: Meta-learners should be simple regularized linear models (Ridge / ElasticNet) to prevent meta-overfitting.
⚠️ Common Interview Pitfalls
- ✕Generating meta-features using training predictions instead of out-of-fold predictions (causes massive target leakage and overfits meta-model).
- ✕Using an overly complex meta-learner (e.g. A 10-layer neural network on meta-predictions, which memorizes base model idiosyncrasies).
🎯 Interviewer Follow-up Inquiries
- ?Why does multi-layer stacking (deep stacking) require nested multi-level OOF splits to prevent hierarchical leakage?
- ?How does Stacking relate to Super Learner in targeted causal learning?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚