GBDT Negative Gradient

GBDT (Gradient Boosting Decision Tree) views boosting as gradient descent in function space: with the additive model $F_M(x) = \sum_{m=1}^{M} \rho_m h_m(x)$, each round updates along the negative gradient of the loss w.r.t. the function values, $F_m = F_{m-1} - \rho_m \nabla_F L\big|_{F = F_{m-1}}$. Round $m$ has three steps: (1) compute the negative-gradient pseudo-residual for each sample, $\tilde{y}_i = -\left[ \frac{\partial L\left(y_i, F(x_i)\right)}{\partial F(x_i)} \right]_{F = F_{m-1}}$; (2) fit a regression tree to $(x_i, \tilde{y}_i)$, obtaining leaf regions $R_{jm}$ ($j = 1, \dots, J_m$); (3) line-search each leaf's output $\rho_{jm} = \arg\min_{\rho} \sum_{x_i \in R_{jm}} L\left(y_i, F_{m-1}(x_i) + \rho\right)$. Under squared loss $L = \frac{1}{2}(y - F)^2$ the pseudo-residual is exactly the residual $\tilde{y}_i = y_i - F_{m-1}(x_i)$ and the leaf output is the leaf-mean; under absolute loss $\tilde{y}_i = \text{sign}\left(y_i - F_{m-1}(x_i)\right)$ and the leaf output is the median (L2 → mean, L1 → median); for binary classification with log loss, $F$ models the log-odds with $\hat{p}_i = \sigma\left(F_{m-1}(x_i)\right)$, pseudo-residual $\tilde{y}_i = y_i - \hat{p}_i$, and leaf values are refined by a Newton step. Training uses shrinkage (learning rate) $\nu \in (0.01, 0.1)$: $F_m = F_{m-1} + \nu \cdot \text{tree}_m$ — each tree contributes less, more trees are needed, but generalization improves substantially; stochastic gradient boosting (subsampling) further reduces variance. Friedman (2001) showed AdaBoost is gradient boosting under exponential loss.

how pseudo-residuals arise, how leaf outputs are determined, differences from AdaBoost, and the role of the learning rate are frequent whiteboard follow-ups; GBDT is also the classic strong baseline for regression, classification, and ranking (LambdaMART).

AdaBoost's exponential loss only fits binary classification and is noise-sensitive — the negative-gradient framework extends boosting to any differentiable loss: each round fits $\frac{\partial L}{\partial F}$ instead of the labels, so adapting a new loss only requires its first derivative (Huber/quantile loss for robust regression, log loss for calibrated probabilities, NDCG surrogates for ranking); fitting residuals directly reduces bias, making it the standard fix for high-bias low-variance weak models (shallow trees), complementing random forests which reduce variance.

Whiteboard the negative-gradient pseudo-residual $\tilde{y}_i = -\left[\frac{\partial L}{\partial F}\right]_{F = F_{m-1}}$: why does it equal $y - F_{m-1}$ under squared loss?

Derive leaf outputs: mean under squared loss, median under absolute loss; why a Newton step for binary log loss?

Relationship to AdaBoost: why is AdaBoost a special case of gradient boosting under exponential loss? How do base learners and weighting differ?

Role of shrinkage $\nu$: why do small learning rates with more trees usually work better? Effect on the bias-variance tradeoff?

Which loss functions can GBDT adapt? Compare its use cases with random forests (bias vs variance reduction)?