Huber Loss & Focal Loss

Huber loss (smooth L1): for residual $r = y - f(x)$, quadratic for small errors and linear for large ones, piecewise defined as $\mathcal{L}_\delta(r) = \begin{cases} \frac{1}{2}r^2, & |r| \le \delta \\ \delta(|r| - \frac{1}{2}\delta), & |r| > \delta \end{cases}$ — continuous and differentiable at $|r| = \delta$, with constant gradient (no explosion) on large outliers while keeping L2's fast convergence on typical samples; in object detection it appears as Smooth L1. Focal Loss: $FL(p_t) = -\alpha_t (1-p_t)^\gamma \log p_t$ where $p_t$ is the predicted probability of the true class, $\gamma \in [0,5]$ is the modulating factor (typically 2) and $\alpha_t$ the class weight — as $p_t \to 1$ (easy samples) the factor $(1-p_t)^\gamma \to 0$ and the loss is sharply down-weighted; as $p_t$ is small (hard samples) the weight is near 1 and the loss is kept. Proposed by RetinaNet to solve “easy negatives drowning the gradient”.

Huber/Smooth L1 — regression with outliers needing stable gradients (bounding-box regression in detection, coordinate regression); Focal — dense object detection and heavily imbalanced classification (background vs object boxes up to 1:1000).

for outliers, MSE's gradient grows linearly with the error and a single anomaly can dominate, while MAE is robust but non-differentiable at zero and converges slowly; Huber compromises — L2 for small errors (fast convergence), L1 gradient clipped at $\pm\delta$ for large ones (robust and differentiable everywhere). Focal solves “easy negatives contribute almost all CE loss and dilute the gradient”: the modulating factor exponentially down-weights easy samples so training focuses on the few hard positives — complementary to, not a replacement for, resampling and class weights.

Test Your Knowledge: Practice Questions for "Huber Loss & Focal Loss"

Single choice pitfall questions with instant feedback and mistake tracking.