M3-064M3: Deep Learning FoundationsLoss Functions & ObjectivesEasy
⚡Flashcard Drill➔Mastery:

Comparing MSE, MAE, and Huber Loss: Robustness and Target Optimality

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-065 →
📐 Mathematical Definition
MSE=1n∑(y−y^)2;MAE=1n∑∣y−y^∣;Huber={12e2∣e∣≤δδ(∣e∣−12δ)∣e∣>δ\text{MSE}=\frac1n\sum(y-\hat y)^2;\quad \text{MAE}=\frac1n\sum|y-\hat y|;\quad \text{Huber}=\begin{cases}\frac12 e^2&|e|\le\delta\\ \delta(|e|-\frac12\delta)&|e|>\delta\end{cases}
⚡ One-Sentence Core Answer
MSE optimizes for the conditional mean and penalizes outliers quadratically; MAE optimizes for the conditional median and is robust; Huber smoothly bridges both regimes.

📌 Key Takeaways

  • •
    MSE: 12(y−y^)2\frac{1}{2}(y - \hat{y})^2; derivative is linear in error, predicting conditional mean E[Y∣X]\mathbb{E}[Y|X]
  • •
    MAE: ∣y−y^∣|y - \hat{y}|; derivative is constant ±1\pm 1, predicting conditional median Median(Y∣X)\text{Median}(Y|X)
  • •
    Huber Loss: quadratic for ∣e∣≤δ|e| \le \delta, linear for ∣e∣>δ|e| > \delta; continuous derivatives with outlier robustness

📐 Mathematical Derivations

Mathematical Formulations and Optimality:
① Mean Squared Error (MSE / L2):
L(y,y^)=12(y−y^)2\mathcal{L}(y, \hat{y}) = \frac{1}{2} (y - \hat{y})^2. Derivative: ∂L∂y^=−(y−y^)\frac{\partial \mathcal{L}}{\partial \hat{y}} = - (y - \hat{y}).
Minimizing expected risk min⁡fE[(Y−f(X))2]\min_f \mathbb{E}[(Y - f(X))^2] yields: f∗(x)=E[Y∣X=x]f^*(x) = \mathbb{E}[Y \mid X = x].
Limitation: Extremely sensitive to outliers; an error of 10 contributes 100×100\times the loss of an error of 1.
② Mean Absolute Error (MAE / L1):
L(y,y^)=∣y−y^∣\mathcal{L}(y, \hat{y}) = |y - \hat{y}|. Derivative: ∂L∂y^=−sign(y−y^)\frac{\partial \mathcal{L}}{\partial \hat{y}} = -\text{sign}(y - \hat{y}).
Minimizing expected risk yields: f∗(x)=Median(Y∣X=x)f^*(x) = \text{Median}(Y \mid X = x).
Limitation: Non-differentiable at y=y^y = \hat{y}; gradient magnitude is constant 1.01.0 even as error approaches zero, causing oscillations around the minimum unless learning rate decays.
③ Huber Loss (Smooth L1):
Lδ(y,y^)={12(y−y^)2if ∣y−y^∣≤δδ(∣y−y^∣−12δ)otherwise\mathcal{L}_\delta(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2 & \text{if } |y - \hat{y}| \le \delta \\ \delta (|y - \hat{y}| - \frac{1}{2}\delta) & \text{otherwise} \end{cases}.
Derivative is continuous: ∂Lδ∂y^={−(y−y^)∣y−y^∣≤δ−δ⋅sign(y−y^)∣y−y^∣>δ\frac{\partial \mathcal{L}_\delta}{\partial \hat{y}} = \begin{cases} - (y - \hat{y}) & |y - \hat{y}| \le \delta \\ - \delta \cdot \text{sign}(y - \hat{y}) & |y - \hat{y}| > \delta \end{cases}. Combines fast quadratic convergence near zero with linear robustness against distant outliers.

🏭 Production Trade-offs

Application domains: Object detection bounding-box regression universally uses Smooth L1 (Huber) to prevent outlier boxes from destabilizing training. Financial forecasting with heavy-tailed returns uses Huber or Quantile loss.
⚠️ Common Interview Pitfalls
  • ✕
    Using MSE in datasets with extreme measurement noise or corrupted labels, causing the model to over-index on outliers
  • ✕
    Using MAE with a high constant learning rate, causing parameters to bounce endlessly across the non-differentiable zero cusp
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does minimizing MSE estimate the conditional mean while MAE estimates the conditional median?
  • ?
    How do you tune the transition threshold parameter δ\delta in Huber loss?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-063: Why Stochastic Gradient Noise Acts as Implicit Regularization in SGD📋 Back to BankNext →M3-065: Cross-Entropy vs Binary Cross-Entropy (BCE) and Numerically Stable Implementations