Loss Function Taxonomy & Gradients

The Loss Function Taxonomy & Gradient Dynamics framework defines the formal mathematical objectives guiding gradient optimization across regression, classification, ranking, and metric learning; the taxonomy spans: 1) Regression: MSE (L2, quadratic penalties sensitive to outliers), MAE (L1, median-robust but non-differentiable at 0), Huber Loss (smooth piecewise combination of L1 and L2); 2) Classification: Binary Cross-Entropy, Multi-class Softmax Cross-Entropy, Focal Loss (modulating factor $\gamma$ suppressing easy-negative gradients); 3) Metric Learning: Triplet Margin Loss and InfoNCE (temperature-scaled contrastive cross-entropy).

Objective function design, robust training against noisy labels, class imbalance mitigation, and contrastive representation learning.

Naive MSE collapses under outlier noise while standard cross-entropy fails under severe class imbalance; tailored loss functions provide robust convexity and optimal parameter convergence.

Derive the gradient of Softmax Cross-Entropy loss with respect to logits $z_i$: $\frac{\partial \mathcal{L}}{\partial z_i} = p_i - y_i$?

Compare Huber Loss vs Smooth L1 Loss in formulation, outlier robustness, and first-derivative continuity?

Explain how Focal Loss's modulating factor $(1 - p_t)^\gamma$ scales down gradients from well-classified easy samples?

Analyze the role of temperature hyperparameter $\tau$ in InfoNCE contrastive loss in balancing hard negative mining against collapse?

Explain how Label Smoothing regularizes cross-entropy by softening one-hot targets and preventing logit overconfidence?