M2-106M2: Classical Machine Learning集成方法 (Bagging/RF)Hard
⚡Flashcard Drill➔Mastery:

Comparing Model Ensembling Strategies: Simple Average, Weighted Average, and Stacking

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-107 →
📐 Mathematical Definition
y^=1m∑ifi(x) ∣ ∑iwifi(x) ∣ Meta(f1(x),…,fm(x))\hat y=\frac1m\sum_i f_i(x)\ \Big|\ \sum_i w_i f_i(x)\ \Big|\ \text{Meta}(f_1(x),\dots,f_m(x))
⚡ One-Sentence Core Answer
Simple average reduces variance without tuning; weighted average optimizes linear coefficients on validation data; Stacking fits a meta-learner on out-of-fold predictions.

📌 Key Takeaways

  • •
    Simple average: equal weights 1/M1/M, robust baseline, effective when models have comparable accuracy
  • •
    Weighted average: weights optimized via constrained regression or Nelder-Mead on hold-out validation set
  • •
    Stacking: uses out-of-fold (OOF) cross-validation predictions as meta-features to train a second-stage learner

📐 Mathematical Derivations

Methodologies:
① Simple Average: y^=1M∑m=1My^m\hat{y} = \frac{1}{M} \sum_{m=1}^M \hat{y}_m. Assumes equal expected error and equal pairwise correlation across base models.
② Weighted Average: y^=∑m=1Mwmy^m\hat{y} = \sum_{m=1}^M w_m \hat{y}_m, subject to ∑wm=1,wm≥0\sum w_m = 1, w_m \ge 0. Weights can be derived analytically by solving min⁡wwTΣw\min_w w^T \Sigma w, where Σij=Cov(ei,ej)\Sigma_{ij} = \text{Cov}(e_i, e_j) is the error covariance matrix of base predictions.
③ Stacking (Wolpert, 1992): To prevent target leakage, split training data into KK folds. For fold kk, train base models on remaining K−1K-1 folds and predict on fold kk. Concatenate these out-of-fold predictions to construct meta-feature matrix Z∈RN×MZ \in \mathbb{R}^{N \times M}. Train a meta-learner (typically Ridge or Logistic Regression) on (Z,y)(Z, y).

🏭 Production Trade-offs

Operational trade-offs: Stacking yields top competition performance but multiplies training and inference latency by the number of base models. In production serving with strict p99 latency SLAs, weighted averaging of 2–3 distinct models is far easier to deploy and maintain.
⚠️ Common Interview Pitfalls
  • ✕
    Generating Stacking meta-features using in-sample training predictions instead of strict out-of-fold cross-validation, causing massive target leakage
  • ✕
    Using an overly complex meta-learner (like deep GBDTs) in Stacking, which easily overfits the low-dimensional meta-features
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is a simple linear model like Ridge regression preferred as the meta-learner in Stacking?
  • ?
    How does multi-layer Stacking (multi-level cascading) guard against compounding target leakage?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-105: The AdaBoost Algorithm and Its Exponential Loss Function📋 Back to BankNext →M2-107: How GBDT Supports Custom Loss Functions via First and Second Order Gradients