M2-106M2: Classical Machine Learning集成方法 (Bagging/RF)Hard
⚡Flashcard Drill➔Mastery:
Comparing Model Ensembling Strategies: Simple Average, Weighted Average, and Stacking
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Simple average reduces variance without tuning; weighted average optimizes linear coefficients on validation data; Stacking fits a meta-learner on out-of-fold predictions.
📌 Key Takeaways
- •Simple average: equal weights , robust baseline, effective when models have comparable accuracy
- •Weighted average: weights optimized via constrained regression or Nelder-Mead on hold-out validation set
- •Stacking: uses out-of-fold (OOF) cross-validation predictions as meta-features to train a second-stage learner
📐 Mathematical Derivations
Methodologies:
① Simple Average: . Assumes equal expected error and equal pairwise correlation across base models.
② Weighted Average: , subject to . Weights can be derived analytically by solving , where is the error covariance matrix of base predictions.
③ Stacking (Wolpert, 1992): To prevent target leakage, split training data into folds. For fold , train base models on remaining folds and predict on fold . Concatenate these out-of-fold predictions to construct meta-feature matrix . Train a meta-learner (typically Ridge or Logistic Regression) on .
① Simple Average: . Assumes equal expected error and equal pairwise correlation across base models.
② Weighted Average: , subject to . Weights can be derived analytically by solving , where is the error covariance matrix of base predictions.
③ Stacking (Wolpert, 1992): To prevent target leakage, split training data into folds. For fold , train base models on remaining folds and predict on fold . Concatenate these out-of-fold predictions to construct meta-feature matrix . Train a meta-learner (typically Ridge or Logistic Regression) on .
🏭 Production Trade-offs
Operational trade-offs: Stacking yields top competition performance but multiplies training and inference latency by the number of base models. In production serving with strict p99 latency SLAs, weighted averaging of 2–3 distinct models is far easier to deploy and maintain.
⚠️ Common Interview Pitfalls
- ✕Generating Stacking meta-features using in-sample training predictions instead of strict out-of-fold cross-validation, causing massive target leakage
- ✕Using an overly complex meta-learner (like deep GBDTs) in Stacking, which easily overfits the low-dimensional meta-features
🎯 Interviewer Follow-up Inquiries
- ?Why is a simple linear model like Ridge regression preferred as the meta-learner in Stacking?
- ?How does multi-layer Stacking (multi-level cascading) guard against compounding target leakage?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚