M2-107M2: Classical Machine Learning梯度提升 (GBDT/XGBoost)Medium
⚡Flashcard Drill➔Mastery:

How GBDT Supports Custom Loss Functions via First and Second Order Gradients

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M2-108 →
📐 Mathematical Definition
gi=∂ℓ(yi,F(xi))∂F,hi=∂2ℓ∂F2g_i=\frac{\partial \ell(y_i,F(x_i))}{\partial F},\qquad h_i=\frac{\partial^2 \ell}{\partial F^2}
⚡ One-Sentence Core Answer
GBDT utilizes second-order Taylor expansion to approximate custom loss functions, deriving optimal leaf values and split gains using only sample gradients and Hessians.

📌 Key Takeaways

  • •
    Taylor expansion: L(y,Ft−1+ft)≈∑[L(y,Ft−1)+gift(xi)+12hift2(xi)]\mathcal{L}(y, F_{t-1} + f_t) \approx \sum [\mathcal{L}(y, F_{t-1}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i)]
  • •
    First and second order derivatives: requires user to supply gi=∂L∂y^ig_i = \frac{\partial \mathcal{L}}{\partial \hat{y}_i} and hi=∂2L∂y^i2h_i = \frac{\partial^2 \mathcal{L}}{\partial \hat{y}_i^2} (hi>0h_i > 0)
  • •
    Closed-form leaf weight: wj∗=−∑i∈Ijgi∑i∈Ijhi+λw_j^* = - \frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda}

📐 Mathematical Derivations

XGBoost formulation for custom objectives: At step tt, expand objective around current prediction y^(t−1)\hat{y}^{(t-1)}:
L~(t)≈∑i=1n[gift(xi)+12hift2(xi)]+γT+12λ∑j=1Twj2\tilde{\mathcal{L}}^{(t)} \approx \sum_{i=1}^n \left[ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) \right] + \gamma T + \frac{1}{2} \lambda \sum_{j=1}^T w_j^2, where gi=[∂ℓ(yi,y^)∂y^]y^(t−1)g_i = \left[\frac{\partial \ell(y_i, \hat{y})}{\partial \hat{y}}\right]_{\hat{y}^{(t-1)}} and hi=[∂2ℓ(yi,y^)∂y^2]y^(t−1)h_i = \left[\frac{\partial^2 \ell(y_i, \hat{y})}{\partial \hat{y}^2}\right]_{\hat{y}^{(t-1)}}.
Let Ij={i∣q(xi)=j}I_j = \{i \mid q(x_i) = j\} be the instance set in leaf jj. Grouping by leaf yields: L~(t)=∑j=1T[(∑i∈Ijgi)wj+12(∑i∈Ijhi+λ)wj2]+γT\tilde{\mathcal{L}}^{(t)} = \sum_{j=1}^T \left[ \left(\sum_{i \in I_j} g_i\right) w_j + \frac{1}{2} \left(\sum_{i \in I_j} h_i + \lambda\right) w_j^2 \right] + \gamma T.
Taking derivative with respect to wjw_j yields the optimal leaf weight: wj∗=−∑i∈Ijgi∑i∈Ijhi+λw_j^* = - \frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda}, with minimum loss: L∗=−12∑j=1T(∑i∈Ijgi)2∑i∈Ijhi+λ+γT\mathcal{L}^* = - \frac{1}{2} \sum_{j=1}^T \frac{\left(\sum_{i \in I_j} g_i\right)^2}{\sum_{i \in I_j} h_i + \lambda} + \gamma T.
Split gain: Gain=12[GL2HL+λ+GR2HR+λ−(GL+GR)2HL+HR+λ]−γ\text{Gain} = \frac{1}{2} \left[ \frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda} \right] - \gamma.

🏭 Production Trade-offs

Implementation requirements: The custom loss must be twice differentiable, and Hessian hih_i must be strictly positive (hi>0h_i > 0, convex loss). For non-smooth objectives like MAE (∣y−y^∣|y - \hat{y}|) where hi=0h_i = 0, approximate using smooth Huber loss or pseudo-Hessians.
⚠️ Common Interview Pitfalls
  • ✕
    Supplying a custom objective where Hessian hi≤0h_i \le 0, causing division by zero or negative gain calculations
  • ✕
    Forgetting that custom objectives operate in logit space for classification, requiring sigmoid transformation to obtain probabilities
🎯 Interviewer Follow-up Inquiries
  • ?
    How do you train XGBoost with an asymmetric custom loss function where under-prediction is penalized 5×5\times more than over-prediction?
  • ?
    Why does XGBoost require second-order Taylor expansion while original Friedman GBDT only used first-order gradients?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM2-106: Comparing Model Ensembling Strategies: Simple Average, Weighted Average, and Stacking📋 Back to BankNext →M2-108: Monotonicity Constraints in GBDT and Their Industrial Business Value