Activation Functions

The evolution of activation functions is the main thread of nonlinearity in deep learning. Sigmoid squashes to $(0,1)$: $\sigma(x)=\frac{1}{1+e^{-x}}$ with derivative $\sigma'(x)=\sigma(x)(1-\sigma(x))=\sigma(1-\sigma)$, peaking at only $0.25$ and saturating at both ends — under deep chain multiplication the gradient inevitably vanishes; Tanh is zero-centered with peak derivative 1 but still saturates. ReLU: $\text{ReLU}(x)=\max(0,x)$ has a constant gradient of 1 on the positive side, fixing vanishing gradients, but zero gradient on the negative side causes Dead ReLU; LeakyReLU patches it with a $0.01x$ slope. GELU weights by the standard normal CDF $\Phi(x)$: $\Phi(x)x+(1-\Phi(x))0.5x$ (often approximated as $x\sigma(1.702x)$), smooth with no hard cutoff — the default for BERT/GPT. SwiGLU brings gating into the FFN: $x\sigma(xW_g)\otimes(xW_u)$, used by LLaMA/Mistral and other modern LLMs to replace ReLU.

Comparison table — Sigmoid: range $(0,1)$, derivative peak $0.25$, both-end saturation; Tanh: zero-centered, derivative peak $1$, still saturates; ReLU: derivative $1$ / $0$, Dead ReLU; LeakyReLU: negative slope $0.01$, mostly fixed; GELU: $x\Phi(x)$, smooth and differentiable everywhere; SwiGLU: $x\sigma(xW_g)\otimes(xW_u)$, gated product, stronger FFN expressivity.

interview classics — “why not Sigmoid in deep nets?”, “why GELU/SwiGLU in GPT?”; frequent asks include deriving $\sigma'=\sigma(1-\sigma)$, comparing saturation ranges and derivative bounds, Dead ReLU causes and fixes, and the ReLU → GELU → SwiGLU motivation in FFN design.

eliminates vanishing gradients and weak expressivity — Sigmoid's peak derivative 0.25 shrinks signals to about $0.25^{50}$ over 50 layers, stalling training; ReLU's constant gradient of 1 lets gradients travel long distances; GELU/Swish smoothness improves optimization; SwiGLU gating is more expressive at equal parameter count — LLaMA reports lower perplexity and more stable training vs ReLU baselines.

Derive $\sigma'(x)=\sigma(x)(1-\sigma(x))$ on the whiteboard; why do deep nets with Sigmoid inevitably suffer vanishing gradients (peak derivative 0.25)?

What does ReLU fix vs Sigmoid? Causes of Dead ReLU and its fixes (LeakyReLU, bias initialization)?

Write the GELU formula $\Phi(x)x+(1-\Phi(x))0.5x$ with its properties; why do BERT/GPT use GELU over ReLU?

Gating mechanism of SwiGLU $x\sigma(xW_g)\otimes(xW_u)$; why do LLaMA/Mistral FFNs use SwiGLU?

Compare derivative ranges and saturation/death issues of Sigmoid/Tanh/ReLU/GELU; why are zero-centered outputs preferred?