E1Easy★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook

Cross-Entropy Loss with Log-Softmax & ignore_index

⚡Flashcard Drill➔

Evaluate cross-entropy safely in log-space via Log-Sum-Exp while masking padded `ignore_index` targets.

⏱️ Time: O(N * C)
💾 Space: O(N * C) 临时指数缓存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def cross_entropy_loss(
logits: np.ndarray, # (B, C) 或 (B, S, C) 未归一化的实数预测
targets: np.ndarray, # (B,) 或 (B, S) 整数真实类别标签
ignore_index: int = -100,
reduction: str = "mean"
) -> float:
# 展平成 2D: (N, C) 与 (N,)
C = logits.shape[-1]
flat_logits =
flat_targets = targets.reshape(-1)
# 1. 计算 Log-Sum-Exp: log(sum(exp(z)))
z_max = np.max(flat_logits, axis=-1, keepdims=True)
exp_shifted = np.exp(flat_logits - z_max)
log_sum_exp = z_max.squeeze(-1) + np.log(np.sum(exp_shifted, axis=-1))
# 2. 提取目标类别的 logits: z_y
# 过滤 ignore_index
valid_mask = flat_targets != ignore_index
valid_targets = flat_targets[valid_mask]
valid_logits = flat_logits[valid_mask]
valid_lse = log_sum_exp[valid_mask]
if len(valid_targets) == 0:
return 0.0
# 获取有效样本的目标 logit: z[i, target[i]]
N_valid = len(valid_targets)
target_logits = valid_logits[np.arange(N_valid), valid_targets]
# 3. 负对数似然损失: L = -(z_y - LSE) = LSE - z_y
losses = valid_lse - target_logits
if reduction == "mean":
return float(np.mean(losses))
elif reduction == "sum":
return
return losses
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does Label Smoothing modify Cross-Entropy Loss, and why does it help calibration?
Next: Binary Cross-Entropy with Logits →
💡

Core Mental Anchor / Mnemonic

Compute LSE minus target logit directly in log-space; normalize only over valid non-ignored tokens.

📐 Mathematical Derivation & Core Formula

L=−log⁡ezy∑jezj=−zy+max⁡(z)+log⁡∑jezj−max⁡(z)\mathcal{L} = -\log \frac{e^{z_y}}{\sum_j e^{z_j}} = -z_y + \max(z) + \log \sum_j e^{z_j - \max(z)}
### Log-Sum-Exp Simplification & ignore_index
Cross-entropy loss is L=−log⁡py\mathcal{L} = -\log p_y. Computing log⁡(softmax(z)y)\log(\mathrm{softmax}(z)_y) directly underflows pyp_y to 0.0 when zy−max⁡z<−88z_y - \max z < -88 (FP32), causing log⁡(0)=−∞\log(0) = -\infty and NaN gradients.
Numerically stable form:
log⁡py=zy−log⁡∑jezj=zy−(max⁡kzk+log⁡∑jezj−max⁡kzk)\log p_y = z_y - \log \sum_j e^{z_j} = z_y - \left(\max_k z_k + \log \sum_j e^{z_j - \max_k z_k}\right)
`ignore_index` handling: Prompt/padding targets marked as -100 are masked out from both the summed loss and the denominator count.

🔄 Tensor Dimensions & Shape Flow

logits: (N, C) -> shifted LSE: (N,) -> gather target_logits: (N_valid,) -> LSE - z_y -> mean/sum over valid tokens

🛡️ Numerical Stability & Pitfalls Checklist

  • Never compute -np.log(softmax(logits)[targets]); tiny probabilities underflow to log(0) = -inf.
  • Divide mean loss by valid token count (targets != ignore_index), never total sequence count N.
  • Backward gradient is simply (softmax(logits) - one_hot(targets)) / N_valid.

💻 Reference Code Implementation

import numpy as np

def cross_entropy_loss(
    logits: np.ndarray,      # (B, C) 或 (B, S, C) 未归一化的实数预测
    targets: np.ndarray,     # (B,) 或 (B, S) 整数真实类别标签
    ignore_index: int = -100,
    reduction: str = "mean"
) -> float:
    # 展平成 2D: (N, C) 与 (N,)
    C = logits.shape[-1]
    flat_logits = logits.reshape(-1, C)
    flat_targets = targets.reshape(-1)
    
    # 1. 计算 Log-Sum-Exp: log(sum(exp(z)))
    z_max = np.max(flat_logits, axis=-1, keepdims=True)
    exp_shifted = np.exp(flat_logits - z_max)
    log_sum_exp = z_max.squeeze(-1) + np.log(np.sum(exp_shifted, axis=-1))
    
    # 2. 提取目标类别的 logits: z_y
    # 过滤 ignore_index
    valid_mask = flat_targets != ignore_index
    valid_targets = flat_targets[valid_mask]
    valid_logits = flat_logits[valid_mask]
    valid_lse = log_sum_exp[valid_mask]
    
    if len(valid_targets) == 0:
        return 0.0
    
    # 获取有效样本的目标 logit: z[i, target[i]]
    N_valid = len(valid_targets)
    target_logits = valid_logits[np.arange(N_valid), valid_targets]
    
    # 3. 负对数似然损失: L = -(z_y - LSE) = LSE - z_y
    losses = valid_lse - target_logits
    
    if reduction == "mean":
        return float(np.mean(losses))
    elif reduction == "sum":
        return float(np.sum(losses))
    return losses

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
logits = np.array([[1000.0, 1001.0, 1002.0], [2.0, 1.0, 0.0]])
targets = np.array([2, -100]) # 第二个被忽略
loss = cross_entropy_loss(logits, targets, ignore_index=-100)
# 样本 0 target 为 2,即最大项,loss 应为 -log(softmax([0, 1, 2])[2]) = 0.4076
assert not np.isnan(loss) and not np.isinf(loss)
assert np.isclose(loss, 0.4076, atol=1e-3)
print("✓ 交叉熵损失数值稳定自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does Label Smoothing modify Cross-Entropy Loss, and why does it help calibration?
It replaces hard one-hot targets with (1−ϵ)+ϵ/C(1 - \epsilon) + \epsilon/C, turning loss into (1−ϵ)LCE+ϵH(u,p)(1 - \epsilon)\mathcal{L}_{\mathrm{CE}} + \epsilon \mathcal{H}(u, p). This discourages unbounded logit margins and improves probability calibration.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: FlashAttention Online Softmax Block-wise AlgorithmBack to Part E & All 105Next: Binary Cross-Entropy with Logits →