E8EasyPart E · Loss Functions Handbook

Huber Loss & Smooth L1 Loss

⚡Flashcard Drill➔

Quadratic L2 behavior near zero and bounded linear L1 behavior for large errors to resist outliers.

⏱️ Time: O(N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def huber_loss(y_true: np.ndarray, y_pred: np.ndarray, delta: float = 1.0) -> float:
"""
参数:
y_true, y_pred: 形状相同的数组
delta: L1 与 L2 的转折阈值
"""
error =
abs_error = np.abs(error)
quadratic = np.minimum(abs_error, delta)
linear = abs_error - quadratic
loss = 0.5 * (quadratic ** 2) + delta * linear
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why have modern object detectors moved from Smooth L1 toward IoU / GIoU / CIoU losses for bounding box regression?
Next: Bradley-Terry Preference Reward Model Loss →
💡

Core Mental Anchor / Mnemonic

Smooth quadratic gradient for small errors; bounded linear slope for outliers.

📐 Mathematical Derivation & Core Formula

Lδ(a)={12a2∣a∣≤δδ(∣a∣−12δ)∣a∣>δ,a=y−y^\mathcal{L}_\delta(a) = \begin{cases} \frac{1}{2} a^2 & |a| \le \delta \\ \delta (|a| - \frac{1}{2} \delta) & |a| > \delta \end{cases}, \quad a = y - \hat{y}
### Quadratic-to-Linear Smooth Transition
- MSE ($L_2$): Smooth everywhere, but (y−y^)2(y - \hat{y})^2 causes large gradients on outliers;
- MAE ($L_1$): Bounded ±1\pm 1 gradient, but non-differentiable at zero and prone to oscillation near the minimum;
- Huber / Smooth $L_1$: Combines 12a2\frac{1}{2}a^2 for small errors ∣a∣≤δ|a| \le \delta with linear δ(∣a∣−12δ)\delta(|a| - \frac{1}{2}\delta) for large errors ∣a∣>δ|a| > \delta, giving bounded gradients against outliers and smooth convergence near zero.

🔄 Tensor Dimensions & Shape Flow

error = pred - true -> abs_error -> piecewise quadratic (<= delta) / linear (> delta) -> mean scalar

🛡️ Numerical Stability & Pitfalls Checklist

  • Bounding-box detectors (Fast/Faster R-CNN) commonly use Smooth L1 with delta=1.0.
  • In PyTorch SmoothL1Loss, transition point delta is configured via the beta parameter.

💻 Reference Code Implementation

import numpy as np

def huber_loss(y_true: np.ndarray, y_pred: np.ndarray, delta: float = 1.0) -> float:
    """
    参数:
        y_true, y_pred: 形状相同的数组
        delta: L1 与 L2 的转折阈值
    """
    error = y_pred - y_true
    abs_error = np.abs(error)
    
    quadratic = np.minimum(abs_error, delta)
    linear = abs_error - quadratic
    
    loss = 0.5 * (quadratic ** 2) + delta * linear
    return float(np.mean(loss))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
y_t = np.array([0.0, 0.0])
y_p = np.array([0.5, 3.0]) # 一个小误差,一个极大异常误差
loss = huber_loss(y_t, y_p, delta=1.0)
# 样本 1: 0.5 * 0.5^2 = 0.125
# 样本 2: 1.0 * (3.0 - 0.5) = 2.5
# 均值: (0.125 + 2.5) / 2 = 1.3125
assert np.isclose(loss, 1.3125)
print("✓ Huber Loss 自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why have modern object detectors moved from Smooth L1 toward IoU / GIoU / CIoU losses for bounding box regression?
Smooth L1 regresses (x,y,w,h)(x, y, w, h) independently without modeling geometric coupling between center, aspect ratio, and overlap area. IoU-family losses optimize box overlap directly and provide scale invariance.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Perplexity (PPL) CalculationBack to Part E & All 105Next: Bradley-Terry Preference Reward Model Loss →