FFN & SwiGLU

the Transformer feed-forward layer $FFN(x) = W_2 \cdot GELU(W_1 x)$ holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: $SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d$ with $SiLU(x) = x \cdot sigmoid(x)$ (Swish), shrinking the intermediate width from $4d$ to $\frac{8}{3}d$ (LLaMA-2 7B: hidden 4096 → intermediate 11008).

every mainstream model after 2023 (LLaMA-2/3, Qwen, Mistral, Gemma) replaced GELU MLPs with SwiGLU; interviews often probe "why gating", "why 8/3 d" and activation memory during training.

plain ReLU/GELU MLPs have no input-conditional selection — information flows the same way for every token; the $4d$ intermediate width inflates parameters and activations; SwiGLU lets different tokens traverse different "sub-networks" via input-dependent gating, improving quality at equal parameter count while actually shrinking parameters and training activation memory ($4d \rightarrow \frac{8}{3}d$).

Why does FFN hold ~2/3 of parameters? Design of intermediate width $4d$ vs $\frac{8}{3}d$?

Essential difference between SwiGLU and a plain GELU MLP (input-dependent gating)?

Properties of SiLU/Swish: why smoother and saturation-free vs ReLU?

GLU family: activations used by ReGLU, GEGLU and SwiGLU?

Why is FFN activation memory large in training, and how to mitigate (caching, recomputation)?