E1Easy★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook
Cross-Entropy Loss with Log-Softmax & ignore_index
Evaluate cross-entropy safely in log-space via Log-Sum-Exp while masking padded `ignore_index` targets.
⏱️ Time:
O(N * C)💾 Space:
O(N * C) 临时指数缓存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def cross_entropy_loss(
logits: np.ndarray, # (B, C) 或 (B, S, C) 未归一化的实数预测
targets: np.ndarray, # (B,) 或 (B, S) 整数真实类别标签
ignore_index: int = -100,
reduction: str = "mean"
) -> float:
# 展平成 2D: (N, C) 与 (N,)
C = logits.shape[-1]
flat_logits =
flat_targets = targets.reshape(-1)
# 1. 计算 Log-Sum-Exp: log(sum(exp(z)))
z_max = np.max(flat_logits, axis=-1, keepdims=True)
exp_shifted = np.exp(flat_logits - z_max)
log_sum_exp = z_max.squeeze(-1) + np.log(np.sum(exp_shifted, axis=-1))
# 2. 提取目标类别的 logits: z_y
# 过滤 ignore_index
valid_mask = flat_targets != ignore_index
valid_targets = flat_targets[valid_mask]
valid_logits = flat_logits[valid_mask]
valid_lse = log_sum_exp[valid_mask]
if len(valid_targets) == 0:
return 0.0
# 获取有效样本的目标 logit: z[i, target[i]]
N_valid = len(valid_targets)
target_logits = valid_logits[np.arange(N_valid), valid_targets]
# 3. 负对数似然损失: L = -(z_y - LSE) = LSE - z_y
losses = valid_lse - target_logits
if reduction == "mean":
return float(np.mean(losses))
elif reduction == "sum":
return
return losses
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does Label Smoothing modify Cross-Entropy Loss, and why does it help calibration?
💡
Core Mental Anchor / Mnemonic
Compute LSE minus target logit directly in log-space; normalize only over valid non-ignored tokens.
📐 Mathematical Derivation & Core Formula
### Log-Sum-Exp Simplification &
Cross-entropy loss is . Computing directly underflows to
Numerically stable form:
`ignore_index` handling: Prompt/padding targets marked as
ignore_indexCross-entropy loss is . Computing directly underflows to
0.0 when (FP32), causing and NaN gradients.Numerically stable form:
`ignore_index` handling: Prompt/padding targets marked as
-100 are masked out from both the summed loss and the denominator count.🔄 Tensor Dimensions & Shape Flow
logits: (N, C) -> shifted LSE: (N,) -> gather target_logits: (N_valid,) -> LSE - z_y -> mean/sum over valid tokens
🛡️ Numerical Stability & Pitfalls Checklist
- Never compute
-np.log(softmax(logits)[targets]); tiny probabilities underflow to log(0) = -inf. - Divide mean loss by valid token count (targets != ignore_index), never total sequence count N.
- Backward gradient is simply (softmax(logits) - one_hot(targets)) / N_valid.
💻 Reference Code Implementation
import numpy as np
def cross_entropy_loss(
logits: np.ndarray, # (B, C) 或 (B, S, C) 未归一化的实数预测
targets: np.ndarray, # (B,) 或 (B, S) 整数真实类别标签
ignore_index: int = -100,
reduction: str = "mean"
) -> float:
# 展平成 2D: (N, C) 与 (N,)
C = logits.shape[-1]
flat_logits = logits.reshape(-1, C)
flat_targets = targets.reshape(-1)
# 1. 计算 Log-Sum-Exp: log(sum(exp(z)))
z_max = np.max(flat_logits, axis=-1, keepdims=True)
exp_shifted = np.exp(flat_logits - z_max)
log_sum_exp = z_max.squeeze(-1) + np.log(np.sum(exp_shifted, axis=-1))
# 2. 提取目标类别的 logits: z_y
# 过滤 ignore_index
valid_mask = flat_targets != ignore_index
valid_targets = flat_targets[valid_mask]
valid_logits = flat_logits[valid_mask]
valid_lse = log_sum_exp[valid_mask]
if len(valid_targets) == 0:
return 0.0
# 获取有效样本的目标 logit: z[i, target[i]]
N_valid = len(valid_targets)
target_logits = valid_logits[np.arange(N_valid), valid_targets]
# 3. 负对数似然损失: L = -(z_y - LSE) = LSE - z_y
losses = valid_lse - target_logits
if reduction == "mean":
return float(np.mean(losses))
elif reduction == "sum":
return float(np.sum(losses))
return losses
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
logits = np.array([[1000.0, 1001.0, 1002.0], [2.0, 1.0, 0.0]])
targets = np.array([2, -100]) # 第二个被忽略
loss = cross_entropy_loss(logits, targets, ignore_index=-100)
# 样本 0 target 为 2,即最大项,loss 应为 -log(softmax([0, 1, 2])[2]) = 0.4076
assert not np.isnan(loss) and not np.isinf(loss)
assert np.isclose(loss, 0.4076, atol=1e-3)
print("✓ 交叉熵损失数值稳定自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does Label Smoothing modify Cross-Entropy Loss, and why does it help calibration?
It replaces hard one-hot targets with , turning loss into . This discourages unbounded logit margins and improves probability calibration.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.