M3-065M3: Deep Learning FoundationsLoss Functions & ObjectivesEasy
⚡Flashcard Drill➔Mastery:

Cross-Entropy vs Binary Cross-Entropy (BCE) and Numerically Stable Implementations

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-066 →
📐 Mathematical Definition
CE=−∑iyilog⁡pi;BCEstable=max⁡(z,0)−z⋅y+log⁡(1+e−∣z∣)\text{CE}=-\sum_i y_i\log p_i;\qquad \text{BCE}_{\text{stable}}=\max(z,0)-z\cdot y+\log(1+e^{-|z|})
⚡ One-Sentence Core Answer
BCE is the 2-class specialization of categorical cross-entropy; numeric stability requires fusing sigmoid/softmax with loss via the Log-Sum-Exp trick to prevent overflow.

📌 Key Takeaways

  • •
    BCE Formulation: L=−[ylog⁡σ(z)+(1−y)log⁡(1−σ(z))]\mathcal{L} = - [y \log \sigma(z) + (1-y) \log(1 - \sigma(z))]
  • •
    Multi-class CE: L=−∑yklog⁡ezk∑ezj\mathcal{L} = - \sum y_k \log \frac{e^{z_k}}{\sum e^{z_j}}
  • •
    Stable formulation: use BCEWithLogitsLoss and CrossEntropyLoss which operate in log-space via log⁡(1+e−∣z∣)\log(1 + e^{-|z|})

📐 Mathematical Derivations

Mathematical Formulations and Stability:
① Binary Cross-Entropy with Logits:
Let p=σ(z)=11+e−zp = \sigma(z) = \frac{1}{1 + e^{-z}}. Then log⁡p=−log⁡(1+e−z)\log p = -\log(1 + e^{-z}) and log⁡(1−p)=−z−log⁡(1+e−z)\log(1-p) = -z - \log(1 + e^{-z}).
Substituting into BCE loss:
L=−[ylog⁡p+(1−y)log⁡(1−p)]=yz+log⁡(1+e−z)−zy=⋯=max⁡(z,0)−zy+log⁡(1+e−∣z∣)\mathcal{L} = - [y \log p + (1-y) \log(1-p)] = y z + \log(1 + e^{-z}) - z y = \dots = \max(z, 0) - z y + \log(1 + e^{-|z|}).
Why stable: Evaluating e−∣z∣e^{-|z|} is guaranteed to never overflow in floating-point because the exponent is strictly ≤0\le 0, bounding the term to (0,1](0, 1].
② Categorical Cross-Entropy (Log-Sum-Exp Trick):
log⁡pk=zk−log⁡∑j=1Kezj\log p_k = z_k - \log \sum_{j=1}^K e^{z_j}.
If zj=100z_j = 100, e100e^{100} overflows to Inf\text{Inf} in FP32. Subtracting c=max⁡jzjc = \max_j z_j preserves exact mathematical equivalence:
log⁡∑j=1Kezj=c+log⁡∑j=1Kezj−c\log \sum_{j=1}^K e^{z_j} = c + \log \sum_{j=1}^K e^{z_j - c}.
Since zj−c≤0z_j - c \le 0, every exponent is non-positive, completely eliminating overflow.

🏭 Production Trade-offs

Implementation rule: Never manually compose torch.log(torch.sigmoid(z)) or torch.log(torch.softmax(z)). Always use fused operators: nn.BCEWithLogitsLoss and nn.CrossEntropyLoss.
⚠️ Common Interview Pitfalls
  • ✕
    Calling sigmoid() on model logits and then passing the result into nn.BCELoss, which causes NaN underflow when predicted probability reaches 0.00.0
  • ✕
    Applying nn.CrossEntropyLoss to logits that have already been passed through nn.Softmax(), applying double softmax
🎯 Interviewer Follow-up Inquiries
  • ?
    How does the Log-Sum-Exp trick maintain numerical stability in mixed precision FP16 calculations?
  • ?
    What is the analytical gradient of fused Softmax + Cross-Entropy with respect to input logit zkz_k?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-064: Comparing MSE, MAE, and Huber Loss: Robustness and Target Optimality📋 Back to BankNext →M3-066: Focal Loss: Design Motivation, Mathematical Derivation, and Modulating Factor