←Back to Science Coding Hub/Part A/A3

A3Medium★ Core Essential · Industrial BedrockPart A · Core Kernels & Activation Functions

Gaussian Error Linear Unit (GELU)

Mastery Status

Stochastic regularizer gating activation based on standard Gaussian CDF, standard in BERT and GPT-2.

⏱️ Time Complexity: O(N)

💾 Space Complexity: O(N)

💡

Core Mental Anchor / Mnemonic

0.5x times (1 + tanh), cubic term scaled by 0.044715

📐 Mathematical Derivation & Core Formula

\mathrm{GELU}(x) = x \Phi(x) \approx 0.5 x \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} (x + 0.044715 x^3)\right)\right)

### Probabilistic Formulation & Approximation
GELU scales input

x

\Phi(x) = P(X \le x)

where

X \sim \mathcal{N}(0, 1)

. Its expectation is

x \Phi(x)

.
Unlike ReLU which zeroes out negative gradients entirely, GELU is smooth everywhere with non-zero negative curvature, easing gradient backpropagation.

\mathrm{GELU}(x) = 0.5 x \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt{2}}\right)\right)

Because

\mathrm{erf}

is compute-intensive, the Tanh polynomial approximation is widely deployed.

🔄 Tensor Dimensions & Shape Flow

(...) -> x^3 -> polynomial inner term -> tanh -> 0.5 * x * (1 + tanh) -> (...)

🛡️ Industrial Numerical Stability & Pitfalls

Memorize the constants: sqrt(2/pi) approx 0.79788456 and cubic coefficient 0.044715
Use fused kernel implementations (fast_gelu) to eliminate intermediate tensor allocations

💻 Industrial Code Implementation

import numpy as np

def gelu_tanh_approx(x: np.ndarray) -> np.ndarray:
    """Classic Tanh approximation (GPT-2 / BERT standard)."""
    const_sqrt_2_pi = np.sqrt(2.0 / np.pi)
    inner = const_sqrt_2_pi * (x + 0.044715 * (x ** 3))
    return 0.5 * x * (1.0 + np.tanh(inner))

def gelu_exact(x: np.ndarray) -> np.ndarray:
    """Exact form using the error function erf."""
    from scipy.special import erf
    return 0.5 * x * (1.0 + erf(x / np.sqrt(2.0)))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter to verify correctness:

import numpy as np
x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
out = gelu_tanh_approx(x)
assert np.isclose(out[2], 0.0), "0 must map to 0"
assert np.isclose(out[3], 0.84119, atol=1e-3), "Value at 1 is inaccurate"
assert out[0] < 0 and out[0] > -0.1, "Negative half-axis should exhibit slight smooth curvature"
print("✓ GELU assertion passed")

🎯 Core Architecture Follow-up Q&A

Q1:Why did modern LLMs (LLaMA, DeepSeek) transition from GELU to SwiGLU?

GELU is a static univariate gating function. SwiGLU uses a bilinear gating mechanism with separate projection weights, dynamically controlling channel pass-through and boosting reasoning performance.

←Prev: Numerically Stable Sigmoid & Log-Sigmoid All 69 Kernels Next: SwiGLU Gated Feed-Forward Network→