Loss Functions

The loss family dictates the geometry of the objective and convergence speed. Cross-entropy is standard for classification: $L=-\sum_{i}y_i\log p_i$, reducing to BCE $-[y\log\hat y+(1-y)\log(1-\hat y)]$ for binary cases; regression uses MSE: $L=\frac{1}{N}\sum\Vert y-\hat y\Vert^2$. The gradient contrast is the interview crux: softmax+CE yields $p_i-y_i$ — linear, never saturating; softmax+MSE yields $(p_i-y_i)p_i(1-p_i)$, where $p(1-p)\to0$ at extreme predictions, killing gradients and slowing learning. For class imbalance, Focal Loss: $FL(p_t)=-\alpha_t(1-p_t)^\gamma\log p_t$; at $\gamma=2$ easy samples are down-weighted by $(1-p_t)^2$. Contrastive learning uses InfoNCE: $-\log\frac{e^{\text{sim}(q,k_+)/\tau}}{\sum_{j=1}^{K}e^{\text{sim}(q,k_j)/\tau}}$, where temperature $\tau$ controls negative-sample pressure; Triplet Loss: $\max(0,\,d(a,p)-d(a,n)+m)$ with margin $m$.

interview favorites — “why CE for classification but MSE for regression?”, “gradient difference between CE and MSE under softmax?”, “how does Focal Loss fix imbalance?”, “how to tune InfoNCE temperature?”; RetinaNet, CLIP/SimCLR and multi-label BCE are common follow-ups.

convergence speed and robustness — CE's $\log$ cancels the softmax denominator to give the linear gradient $p-y$, while MSE's gradient vanishes under $p(1-p)$ in saturated regions, so CE converges up to an order of magnitude faster on the same net; Focal down-weights abundant easy samples via $(1-p_t)^\gamma$, making RetinaNet the first one-stage detector to beat two-stage on COCO; InfoNCE's $\tau$ sets the difficulty weight of negatives and directly determines CLIP/SimCLR embedding quality.

Test Your Knowledge: Practice Questions for "Loss Functions"

Single choice pitfall questions with instant feedback and mistake tracking.