M2-107M2: Classical Machine Learning梯度提升 (GBDT/XGBoost)Medium
⚡Flashcard Drill➔Mastery:
How GBDT Supports Custom Loss Functions via First and Second Order Gradients
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
GBDT utilizes second-order Taylor expansion to approximate custom loss functions, deriving optimal leaf values and split gains using only sample gradients and Hessians.
📌 Key Takeaways
- •Taylor expansion:
- •First and second order derivatives: requires user to supply and ()
- •Closed-form leaf weight:
📐 Mathematical Derivations
XGBoost formulation for custom objectives: At step , expand objective around current prediction :
, where and .
Let be the instance set in leaf . Grouping by leaf yields: .
Taking derivative with respect to yields the optimal leaf weight: , with minimum loss: .
Split gain: .
, where and .
Let be the instance set in leaf . Grouping by leaf yields: .
Taking derivative with respect to yields the optimal leaf weight: , with minimum loss: .
Split gain: .
🏭 Production Trade-offs
Implementation requirements: The custom loss must be twice differentiable, and Hessian must be strictly positive (, convex loss). For non-smooth objectives like MAE () where , approximate using smooth Huber loss or pseudo-Hessians.
⚠️ Common Interview Pitfalls
- ✕Supplying a custom objective where Hessian , causing division by zero or negative gain calculations
- ✕Forgetting that custom objectives operate in logit space for classification, requiring sigmoid transformation to obtain probabilities
🎯 Interviewer Follow-up Inquiries
- ?How do you train XGBoost with an asymmetric custom loss function where under-prediction is penalized more than over-prediction?
- ?Why does XGBoost require second-order Taylor expansion while original Friedman GBDT only used first-order gradients?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚