A3Medium★ Core Essential · Industrial BedrockPart A · Core Kernels & Activation Functions

Gaussian Error Linear Unit (GELU)

⚡Flashcard Drill➔

Stochastic regularizer gating activation based on standard Gaussian CDF, standard in BERT and GPT-2.

⏱️ Time: O(N)
💾 Space: O(N)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def gelu_tanh_approx(x: np.ndarray) -> np.ndarray:
"""Classic Tanh approximation (GPT-2 / BERT standard)."""
const_sqrt_2_pi =
inner = const_sqrt_2_pi * (x + 0.044715 * (x ** 3))
return 0.5 * x * (1.0 + np.tanh(inner))
def gelu_exact(x: np.ndarray) -> np.ndarray:
"""Exact form using the error function erf."""
from scipy.special import erf
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why did modern LLMs (LLaMA, DeepSeek) transition from GELU to SwiGLU?
Next: SwiGLU Gated Feed-Forward Network →
💡

Core Mental Anchor / Mnemonic

0.5x times (1 + tanh), cubic term scaled by 0.044715

📐 Mathematical Derivation & Core Formula

GELU(x)=xΦ(x)≈0.5x(1+tanh⁡(2π(x+0.044715x3)))\mathrm{GELU}(x) = x \Phi(x) \approx 0.5 x \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} (x + 0.044715 x^3)\right)\right)
### Probabilistic Formulation & Approximation
GELU scales input xx by Φ(x)=P(X≤x)\Phi(x) = P(X \le x) where X∼N(0,1)X \sim \mathcal{N}(0, 1). Its expectation is xΦ(x)x \Phi(x).
Unlike ReLU which zeroes out negative gradients entirely, GELU is smooth everywhere with non-zero negative curvature, easing gradient backpropagation.
GELU(x)=0.5x(1+erf(x2))\mathrm{GELU}(x) = 0.5 x \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt{2}}\right)\right)
Because erf\mathrm{erf} is compute-intensive, the Tanh polynomial approximation is widely deployed.

🔄 Tensor Dimensions & Shape Flow

(...) -> x^3 -> polynomial inner term -> tanh -> 0.5 * x * (1 + tanh) -> (...)

🛡️ Numerical Stability & Pitfalls Checklist

  • Memorize the constants: sqrt(2/pi) approx 0.79788456 and cubic coefficient 0.044715
  • Use fused kernel implementations (fast_gelu) to eliminate intermediate tensor allocations

💻 Reference Code Implementation

import numpy as np

def gelu_tanh_approx(x: np.ndarray) -> np.ndarray:
    """Classic Tanh approximation (GPT-2 / BERT standard)."""
    const_sqrt_2_pi = np.sqrt(2.0 / np.pi)
    inner = const_sqrt_2_pi * (x + 0.044715 * (x ** 3))
    return 0.5 * x * (1.0 + np.tanh(inner))

def gelu_exact(x: np.ndarray) -> np.ndarray:
    """Exact form using the error function erf."""
    from scipy.special import erf
    return 0.5 * x * (1.0 + erf(x / np.sqrt(2.0)))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
x = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])
out = gelu_tanh_approx(x)
assert np.isclose(out[2], 0.0), "0 must map to 0"
assert np.isclose(out[3], 0.84119, atol=1e-3), "Value at 1 is inaccurate"
assert out[0] < 0 and out[0] > -0.1, "Negative half-axis should exhibit slight smooth curvature"
print("✓ GELU assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why did modern LLMs (LLaMA, DeepSeek) transition from GELU to SwiGLU?
GELU is a static univariate gating function. SwiGLU uses a bilinear gating mechanism with separate projection weights, dynamically controlling channel pass-through and boosting reasoning performance.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Numerically Stable Sigmoid & Log-SigmoidBack to Part A & All 105Next: SwiGLU Gated Feed-Forward Network →