E2Easy★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook

Binary Cross-Entropy with Logits

⚡Flashcard Drill➔

Fuse Sigmoid and BCE via max(x, 0) - x*y + log1p(exp(-|x|)) to prevent overflow and log(0) underflow.

⏱️ Time: O(N) 逐元素计算
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def bce_with_logits_loss(logits: np.ndarray, targets: np.ndarray, reduction: str = "mean") -> float:
"""
参数:
logits: 实数预测,形状任意,如 (N,)
targets: 真实标签 0.0 或 1.0,形状与 logits 严格一致
"""
# 核心公式: max(x, 0) - x * y + log1p(exp(-abs(x)))
max_part =
neg_abs = -np.abs(logits)
loss = max_part - logits * targets + np.log1p(np.exp(neg_abs))
if reduction == "mean":
return float(np.mean(loss))
elif reduction == "sum":
return
return loss
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How can pos_weight be incorporated into BCEWithLogitsLoss under severe class imbalance (e.g., 1% positives)?
Next: Focal Loss for Dense Object Detection →
💡

Core Mental Anchor / Mnemonic

Use max(x, 0) - x*y + log1p(exp(-|x|)) so exponents stay non-positive everywhere.

📐 Mathematical Derivation & Core Formula

L=max⁡(x,0)−x⋅y+log⁡(1+e−∣x∣)\mathcal{L} = \max(x, 0) - x \cdot y + \log(1 + e^{-|x|})
### Closed-Form Numerical Stabilization for BCE With Logits
Starting from L=−ylog⁡σ(x)−(1−y)log⁡(1−σ(x))\mathcal{L} = -y \log \sigma(x) - (1 - y) \log(1 - \sigma(x)) and substituting σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}:
L=(1−y)x+log⁡(1+e−x)=x−xy+log⁡(1+e−x)\mathcal{L} = (1 - y)x + \log(1 + e^{-x}) = x - xy + \log(1 + e^{-x})
Rewriting log⁡(1+e−x)=max⁡(x,0)−x+log⁡(1+e−∣x∣)\log(1 + e^{-x}) = \max(x, 0) - x + \log(1 + e^{-|x|}) yields the compact stable identity:
L=max⁡(x,0)−x⋅y+log⁡(1+e−∣x∣)\mathcal{L} = \max(x, 0) - x \cdot y + \log(1 + e^{-|x|})
Since −∣x∣≤0-|x| \le 0, e−∣x∣∈(0,1]e^{-|x|} \in (0, 1], preventing exponential overflow across all real inputs.

🔄 Tensor Dimensions & Shape Flow

logits x, targets y: (...) -> max(x, 0) - x*y + log1p(exp(-abs(x))) -> mean scalar

🛡️ Numerical Stability & Pitfalls Checklist

  • Use np.log1p(np.exp(-np.abs(x))) to maintain precision when exp(-|x|) is very small.
  • Negate abs(x) so the exponent argument is always <= 0, keeping exp bounded in (0, 1].
  • Gradient with respect to logits is simply sigmoid(x) - y.

💻 Reference Code Implementation

import numpy as np

def bce_with_logits_loss(logits: np.ndarray, targets: np.ndarray, reduction: str = "mean") -> float:
    """
    参数:
        logits: 实数预测,形状任意,如 (N,)
        targets: 真实标签 0.0 或 1.0,形状与 logits 严格一致
    """
    # 核心公式: max(x, 0) - x * y + log1p(exp(-abs(x)))
    max_part = np.maximum(logits, 0.0)
    neg_abs = -np.abs(logits)
    loss = max_part - logits * targets + np.log1p(np.exp(neg_abs))
    
    if reduction == "mean":
        return float(np.mean(loss))
    elif reduction == "sum":
        return float(np.sum(loss))
    return loss

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
logits = np.array([-1000.0, 0.0, 1000.0])
targets = np.array([0.0, 1.0, 1.0])
loss = bce_with_logits_loss(logits, targets)
assert not np.isnan(loss) and not np.isinf(loss)
# logits=-1000, target=0 时预测极准,loss 应趋近于 0
assert np.isclose(bce_with_logits_loss(np.array([-1000.0]), np.array([0.0])), 0.0)
print("✓ BCE with Logits 数值稳定自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How can pos_weight be incorporated into BCEWithLogitsLoss under severe class imbalance (e.g., 1% positives)?
Multiply the positive loss contribution by ww: L=(1−y)x+(1+(w−1)y)log⁡(1+e−x)\mathcal{L} = (1 - y)x + (1 + (w - 1)y)\log(1 + e^{-x}) (or stabilized with max⁡(−x,0)\max(-x, 0)). When y=1y=1, gradients scale by ww, counteracting majority negative dominance.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Cross-Entropy Loss with Log-Softmax & ignore_indexBack to Part E & All 105Next: Focal Loss for Dense Object Detection →