M2-027M2: Classical Machine Learning集成方法 (Bagging/RF)Easy
⚡Flashcard Drill➔Mastery:

Compare the Core Theoretical and Architectural Differences Between Bagging and Boosting

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-028 →
📐 Mathematical Definition
Bagging: f^=1B∑bfb;Boosting: f^=∑tαtht\text{Bagging}:\ \hat f=\frac1B\sum_b f_b;\quad \text{Boosting}:\ \hat f=\sum_t\alpha_t h_t
⚡ One-Sentence Core Answer
Bagging builds independent, deep trees in parallel on bootstrap samples to reduce variance; Boosting builds sequential, shallow trees iteratively on gradient residuals to reduce bias.

📌 Key Takeaways

  • •
    Training Paradigm: Bagging is parallel and independent; Boosting is sequential and dependent (TmT_m corrects errors of Tm−1T_{m-1}).
  • •
    Base Learners: Bagging uses high-capacity, low-bias deep learners (unpruned trees); Boosting uses low-capacity, low-variance weak learners (shallow stumps).
  • •
    Primary Mechanism: Bagging averages predictions to reduce variance: Var→ρσ2+1−ρBσ2\text{Var} \to \rho \sigma^2 + \frac{1-\rho}{B}\sigma^2; Boosting performs gradient descent in function space to reduce bias.

📐 Mathematical Derivations

Variance reduction in Bagging: Let BB trees have individual prediction variance σ2\sigma^2 with pairwise correlation ρ\rho. The variance of the ensemble average fˉ=1B∑b=1Bfb\bar{f} = \frac{1}{B}\sum_{b=1}^B f_b is: Var(fˉ)=Var(1B∑fb)=1B2[∑b=1BVar(fb)+∑i≠jCov(fi,fj)]=1B2[Bσ2+B(B−1)ρσ2]=ρσ2+1−ρBσ2\text{Var}(\bar{f}) = \text{Var}\left(\frac{1}{B}\sum f_b\right) = \frac{1}{B^2}\left[\sum_{b=1}^B \text{Var}(f_b) + \sum_{i \ne j} \text{Cov}(f_i, f_j)\right] = \frac{1}{B^2}[B\sigma^2 + B(B-1)\rho\sigma^2] = \rho\sigma^2 + \frac{1-\rho}{B}\sigma^2. As B→∞B \to \infty, the second term vanishes, leaving ρσ2\rho\sigma^2. This proves that the performance bound of Bagging is dictated entirely by tree correlation ρ\rho, motivating Random Forest feature subsampling to drive ρ→0\rho \to 0. In Boosting, the sequential update is Fm(x)=Fm−1(x)+ηhm(x)F_m(x) = F_{m-1}(x) + \eta h_m(x) where hm=arg⁡min⁡h∑L(yi,Fm−1(xi)+h(xi))h_m = \arg\min_h \sum L(y_i, F_{m-1}(x_i) + h(x_i)), shrinking residual bias exponentially.

🏭 Production Trade-offs

Industrial selection: (1) Random Forest (Bagging): Highly robust to hyperparameters, impossible to overfit by simply adding more trees, embarrassingly parallel, ideal for noisy datasets. (2) GBDT / LightGBM (Boosting): Consistently achieves higher predictive accuracy on clean tabular benchmarks, but requires careful tuning of learning rate η\eta, tree depth, and early stopping to prevent overfitting.
⚠️ Common Interview Pitfalls
  • ✕
    Growing shallow trees in Random Forest (Bagging requires low-bias base learners; shallow trees cause severe underfitting).
  • ✕
    Setting learning rate too high in GBDT without early stopping, causing rapid overfitting.
🎯 Interviewer Follow-up Inquiries
  • ?
    Why can Random Forest never overfit simply by increasing the number of trees BB?
  • ?
    How does Gradient Boosting interpret sequential tree fitting as functional gradient descent in L2L_2 function space?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-026: Compare Industrial Strategies for Handling Missing Values and Categorical Features in Tree Models📋 Back to BankNext →M2-028: Explain the Two Sources of Randomness in Random Forests and How They Decorrelate Trees