Activation Functions

The evolution of activation functions is the main thread of nonlinearity in deep learning. Sigmoid squashes to $(0,1)$: $\sigma(x)=\frac{1}{1+e^{-x}}$ with derivative $\sigma'(x)=\sigma(x)(1-\sigma(x))=\sigma(1-\sigma)$, peaking at only $0.25$ and saturating at both ends — under deep chain multiplication the gradient inevitably vanishes; Tanh is zero-centered with peak derivative 1 but still saturates. ReLU: $\text{ReLU}(x)=\max(0,x)$ has a constant gradient of 1 on the positive side, fixing vanishing gradients, but zero gradient on the negative side causes Dead ReLU; LeakyReLU patches it with a $0.01x$ slope. GELU weights by the standard normal CDF $\Phi(x)$: $\Phi(x)x+(1-\Phi(x))0.5x$ (often approximated as $x\sigma(1.702x)$), smooth with no hard cutoff — the default for BERT/GPT. SwiGLU brings gating into the FFN: $x\sigma(xW_g)\otimes(xW_u)$, used by LLaMA/Mistral and other modern LLMs to replace ReLU.

Comparison table — Sigmoid: range $(0,1)$, derivative peak $0.25$, both-end saturation; Tanh: zero-centered, derivative peak $1$, still saturates; ReLU: derivative $1$ / $0$, Dead ReLU; LeakyReLU: negative slope $0.01$, mostly fixed; GELU: $x\Phi(x)$, smooth and differentiable everywhere; SwiGLU: $x\sigma(xW_g)\otimes(xW_u)$, gated product, stronger FFN expressivity.

interview classics — “why not Sigmoid in deep nets?”, “why GELU/SwiGLU in GPT?”; frequent asks include deriving $\sigma'=\sigma(1-\sigma)$, comparing saturation ranges and derivative bounds, Dead ReLU causes and fixes, and the ReLU → GELU → SwiGLU motivation in FFN design.

eliminates vanishing gradients and weak expressivity — Sigmoid's peak derivative 0.25 shrinks signals to about $0.25^{50}$ over 50 layers, stalling training; ReLU's constant gradient of 1 lets gradients travel long distances; GELU/Swish smoothness improves optimization; SwiGLU gating is more expressive at equal parameter count — LLaMA reports lower perplexity and more stable training vs ReLU baselines.

Test Your Knowledge: Practice Questions for "Activation Functions"

Single choice pitfall questions with instant feedback and mistake tracking.