💬 Discord
Sign In
←Back to Science Coding Hub/Part A/A2
A2Easy★ Core Essential · Industrial BedrockPart A · Core Kernels & Activation Functions

Numerically Stable Sigmoid & Log-Sigmoid

Piecewise formulation avoiding overflow on large negatives; Log-Sigmoid powered by logaddexp for robust BCE training.

⏱️ Time Complexity: O(N)
💾 Space Complexity: O(N)
💡

Core Mental Anchor / Mnemonic

Standard form for positives, multiply exp(z) for negatives; use logaddexp for log-sigmoid

📐 Mathematical Derivation & Core Formula

σ(z)=11+e−z,log⁡σ(z)=−log⁡(1+e−z)=−logaddexp(0,−z)\sigma(z) = \frac{1}{1 + e^{-z}}, \quad \log \sigma(z) = -\log(1 + e^{-z}) = -\mathrm{logaddexp}(0, -z)
### Numerical Defense & Formulation
When z≪0z \ll 0, −z≫0-z \gg 0, directly evaluating e−ze^{-z} overflows.
For z<0z < 0, multiply numerator and denominator by eze^z:
σ(z)=ez1+ez\sigma(z) = \frac{e^z}{1 + e^z}
Because ez∈(0,1]e^z \in (0, 1], overflow is completely avoided.

For log⁡σ(z)\log \sigma(z), negative saturation yields log⁡(0)=−∞\log(0) = -\infty. We use the exact identity −log⁡(1+e−z)=−logaddexp(0,−z)-\log(1 + e^{-z}) = -\mathrm{logaddexp}(0, -z) to retain full precision.

🔄 Tensor Dimensions & Shape Flow

(...) -> partition into pos/neg masks -> piecewise exp -> combine back to (...)

🛡️ Industrial Numerical Stability & Pitfalls

  • Do not evaluate 1.0 / (1.0 + np.exp(-z)) unconditionally
  • Never compute Log-Sigmoid as np.log(sigmoid(z)); use np.logaddexp(0.0, -z)
  • Sigmoid derivative satisfies d_sigmoid = s * (1.0 - s)

💻 Industrial Code Implementation

import numpy as np

def stable_sigmoid(z: np.ndarray) -> np.ndarray:
    """Piecewise evaluation to prevent overflow for large negative z."""
    out = np.empty_like(z, dtype=np.float64)
    pos_mask = z >= 0
    out[pos_mask] = 1.0 / (1.0 + np.exp(-z[pos_mask]))
    exp_z = np.exp(z[~pos_mask])
    out[~pos_mask] = exp_z / (1.0 + exp_z)
    return out

def stable_log_sigmoid(z: np.ndarray) -> np.ndarray:
    """Stable log(sigmoid(z)) using logaddexp."""
    return -np.logaddexp(0.0, -z)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter to verify correctness:

import numpy as np
z = np.array([-1000.0, 0.0, 1000.0])
s = stable_sigmoid(z)
assert s[0] == 0.0 and s[1] == 0.5 and s[2] == 1.0
assert not np.isnan(s).any()
ls = stable_log_sigmoid(z)
assert not np.isneginf(ls[0]), "Extreme negatives should not return -inf"
assert np.isclose(ls[0], -1000.0), f"Actual: {ls[0]}"
print("✓ Sigmoid assertion passed")

🎯 Core Architecture Follow-up Q&A

Q1:Why fuse Sigmoid and BCE into a single BCEWithLogitsLoss operator?
Fusing them eliminates intermediate activation round-trips to HBM and computes max⁡(x,0)−xy+log⁡(1+e−∣x∣)\max(x, 0) - xy + \log(1 + e^{-|x|}), preventing gradient vanishing and log(0) errors.
←Prev: Numerically Stable SoftmaxAll 69 KernelsNext: Gaussian Error Linear Unit (GELU)→