A1Easy★ Core Essential · Practice FoundationPart A · Core Kernels & Activation Functions
Numerically Stable Softmax
Numerically stable implementation subtracting the axis maximum to prevent exponent overflow while preserving dimensional broadcasting.
⏱️ Time:
O(N * C)💾 Space:
O(N * C)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
"""
Numerically stable Softmax implementation.
Subtracts the maximum value along the target axis before exponentiation to prevent overflow.
"""
x_max =
exp_shifted = np.exp(x - x_max)
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What happens numerically if all inputs are very negative (e.g., [-10000, -10000] vs. [-10000, -11000])?
Q2: Why is LogSoftmax numerically superior to log(softmax(x))?
💡
Core Mental Anchor / Mnemonic
Subtract max to guard against overflow; keepdims ensures broadcasting; denominator has 1 to avoid zero-division
📐 Mathematical Derivation & Core Formula
### Mathematical Principles & Derivation
Softmax maps real-valued logits to a valid probability distribution. Directly evaluating triggers floating-point overflow for (FP64) or (FP32), yielding
Using the shift-invariance identity:
Setting guarantees that , bounding all exponents into . The denominator is always , eliminating division by zero.
Softmax maps real-valued logits to a valid probability distribution. Directly evaluating triggers floating-point overflow for (FP64) or (FP32), yielding
NaN or inf.Using the shift-invariance identity:
Setting guarantees that , bounding all exponents into . The denominator is always , eliminating division by zero.
🔄 Tensor Dimensions & Shape Flow
(B, S, C) -> np.max(keepdims=True) -> (B, S, 1) -> broadcast subtraction -> (B, S, C) -> exp -> (B, S, C) -> sum -> (B, S, 1) -> broadcast division -> (B, S, C)
🛡️ Numerical Stability & Pitfalls Checklist
- Must subtract max(x, keepdims=True) prior to exp; never call np.exp(x) directly
- Keep keepdims=True on both max and sum for correct multi-dimensional broadcasting
- In backpropagation, use the closed-form dlogits = probs - labels instead of manual chain rule
💻 Reference Code Implementation
import numpy as np
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
"""
Numerically stable Softmax implementation.
Subtracts the maximum value along the target axis before exponentiation to prevent overflow.
"""
x_max = np.max(x, axis=axis, keepdims=True)
exp_shifted = np.exp(x - x_max)
return exp_shifted / np.sum(exp_shifted, axis=axis, keepdims=True)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
x = np.array([[1000.0, 1001.0, 1002.0], [-1000.0, -1001.0, -1002.0]])
probs = softmax(x, axis=-1)
assert not np.isnan(probs).any(), "Contains NaN!"
assert np.allclose(probs.sum(axis=-1), [1.0, 1.0]), "Probabilities must sum to 1"
assert np.isclose(probs[0, 2], 0.66524096), "Numerical error exceeds tolerance"
print("✓ Softmax assertion passed")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What happens numerically if all inputs are very negative (e.g., [-10000, -10000] vs. [-10000, -11000])?
Behavior depends on the relative gap , not the negative magnitude itself: for , subtracting the max shifts inputs to , yielding rather than a one-hot vector. Smaller terms underflow to
0 only when exceeds the floating-point underflow threshold (~745 in FP64, ~104 in FP32, such as ). Because the maximum term always contributes , the denominator stays at least 1, avoiding the 0/0 = NaN failure of unshifted exp(x).Q2:Why is LogSoftmax numerically superior to log(softmax(x))?
Evaluating log(softmax(x)) fails if softmax probabilities underflow to 0. LogSoftmax is computed as , operating strictly in log-space.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.