E1Easy★ Core Essential · Industrial BedrockPart E · Loss Functions Handbook
Cross-Entropy Loss with Log-Softmax & ignore_index
Industrial-grade implementation and mathematical foundations of Cross-Entropy Loss with Log-Softmax & ignore_index.
⏱️ Time Complexity:
O(N * C)💾 Space Complexity:
O(N * C) 临时指数缓存💡
Core Mental Anchor / Mnemonic
Master Cross-Entropy Loss with Log-Softmax & ignore_index: enforce numerical stability, check tensor shapes, and eliminate redundant memory allocations.
📐 Mathematical Derivation & Core Formula
### Mathematical Derivation & Theoretical Principles
Detailed first-principles formulation and architectural mechanics for Cross-Entropy Loss with Log-Softmax & ignore_index.
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
Detailed first-principles formulation and architectural mechanics for Cross-Entropy Loss with Log-Softmax & ignore_index.
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
🔄 Tensor Dimensions & Shape Flow
logits: (N, C) -> max 减法与 exp -> LSE: (N,) -> 索引抽取 target_logits: (N_valid,) -> LSE - z_y -> (N_valid,) -> mean/sum 标量
🛡️ Industrial Numerical Stability & Pitfalls
- Ensure proper multi-dimensional tensor broadcasting and keepdims retention.
- Enforce numerical guards (eps clamping and overflow thresholds) during exponentiation and division.
- Verify train versus eval mode behavioral distinctions (e.g. frozen running statistics and dropout bypass).
💻 Industrial Code Implementation
import numpy as np
def cross_entropy_loss(
logits: np.ndarray, # (B, C) 或 (B, S, C) 未归一化的实数预测
targets: np.ndarray, # (B,) 或 (B, S) 整数真实类别标签
ignore_index: int = -100,
reduction: str = "mean"
) -> float:
# 展平成 2D: (N, C) 与 (N,)
C = logits.shape[-1]
flat_logits = logits.reshape(-1, C)
flat_targets = targets.reshape(-1)
# 1. 计算 Log-Sum-Exp: log(sum(exp(z)))
z_max = np.max(flat_logits, axis=-1, keepdims=True)
exp_shifted = np.exp(flat_logits - z_max)
log_sum_exp = z_max.squeeze(-1) + np.log(np.sum(exp_shifted, axis=-1))
# 2. 提取目标类别的 logits: z_y
# 过滤 ignore_index
valid_mask = flat_targets != ignore_index
valid_targets = flat_targets[valid_mask]
valid_logits = flat_logits[valid_mask]
valid_lse = log_sum_exp[valid_mask]
if len(valid_targets) == 0:
return 0.0
# 获取有效样本的目标 logit: z[i, target[i]]
N_valid = len(valid_targets)
target_logits = valid_logits[np.arange(N_valid), valid_targets]
# 3. 负对数似然损失: L = -(z_y - LSE) = LSE - z_y
losses = valid_lse - target_logits
if reduction == "mean":
return float(np.mean(losses))
elif reduction == "sum":
return float(np.sum(losses))
return losses
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter to verify correctness:
import numpy as np
logits = np.array([[1000.0, 1001.0, 1002.0], [2.0, 1.0, 0.0]])
targets = np.array([2, -100]) # 第二个被忽略
loss = cross_entropy_loss(logits, targets, ignore_index=-100)
# 样本 0 target 为 2，即最大项，loss 应为 -log(softmax([0, 1, 2])[2]) = 0.4076
assert not np.isnan(loss) and not np.isinf(loss)
assert np.isclose(loss, 0.4076, atol=1e-3)
print("✓ 交叉熵损失数值稳定自测通过")🎯 Core Architecture Follow-up Q&A
Q1:What are the key trade-offs and memory bottlenecks when deploying Cross-Entropy Loss with Log-Softmax & ignore_index in high-throughput inference?
Memory bandwidth (HBM to SRAM I/O) is the primary latency factor. Fusing element-wise operations and avoiding intermediate tensor materialization significantly outperforms naive implementations.
Q2:How does Cross-Entropy Loss with Log-Softmax & ignore_index handle extreme numerical boundaries or precision reduction (FP16/BF16/INT8)?
Under low precision, operations must be upcasted to FP32 during accumulation to prevent underflow/overflow, followed by proper scaling and clamping before converting back to the target format.