⚡ LLMsID: feedforward-swiglu

FFN & SwiGLU

Q: Why does FFN hold ~2/3 of parameters? Design of intermediate width $4d$ vs $\frac{8}{3}d$?

【FFN & SwiGLU 核心考点 1】Why does FFN hold ~2/3 of parameters? Design of intermediate width $4d$ vs $\frac{8}{3}d$?。技术解析：Definition: the Transformer feed-forward layer FFN(x) = W_2 \cdot GELU(W_1 x) holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d with SiLU(x) = x \cdot sigmoid(x) (Swish), shrinking the intermedi... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/feedforward-swiglu

Q: Essential difference between SwiGLU and a plain GELU MLP (input-dependent gating)?

【FFN & SwiGLU 核心考点 2】Essential difference between SwiGLU and a plain GELU MLP (input-dependent gating)?。技术解析：Definition: the Transformer feed-forward layer FFN(x) = W_2 \cdot GELU(W_1 x) holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d with SiLU(x) = x \cdot sigmoid(x) (Swish), shrinking the intermedi... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/feedforward-swiglu

Q: Properties of SiLU/Swish: why smoother and saturation-free vs ReLU?

【FFN & SwiGLU 核心考点 3】Properties of SiLU/Swish: why smoother and saturation-free vs ReLU?。技术解析：Definition: the Transformer feed-forward layer FFN(x) = W_2 \cdot GELU(W_1 x) holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d with SiLU(x) = x \cdot sigmoid(x) (Swish), shrinking the intermedi... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/feedforward-swiglu

Q: GLU family: activations used by ReGLU, GEGLU and SwiGLU?

【FFN & SwiGLU 核心考点 4】GLU family: activations used by ReGLU, GEGLU and SwiGLU?。技术解析：Definition: the Transformer feed-forward layer FFN(x) = W_2 \cdot GELU(W_1 x) holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d with SiLU(x) = x \cdot sigmoid(x) (Swish), shrinking the intermedi... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/feedforward-swiglu

Q: Why is FFN activation memory large in training, and how to mitigate (caching, recomputation)?

【FFN & SwiGLU 核心考点 5】Why is FFN activation memory large in training, and how to mitigate (caching, recomputation)?。技术解析：Definition: the Transformer feed-forward layer FFN(x) = W_2 \cdot GELU(W_1 x) holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation: SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d with SiLU(x) = x \cdot sigmoid(x) (Swish), shrinking the intermedi... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/feedforward-swiglu

前馈层与激活

🎯Core Definition

the Transformer feed-forward layer

FFN(x) = W_2 \cdot GELU(W_1 x)

holds roughly 2/3 of model parameters (excluding embeddings); SwiGLU is a gated activation:

SwiGLU(x) = (SiLU(x W_g)) \cdot (x W_u) W_d

with

SiLU(x) = x \cdot sigmoid(x)

(Swish), shrinking the intermediate width from

4d

\frac{8}{3}d

(LLaMA-2 7B: hidden 4096 → intermediate 11008).

💡Use Cases

every mainstream model after 2023 (LLaMA-2/3, Qwen, Mistral, Gemma) replaced GELU MLPs with SwiGLU; interviews often probe "why gating", "why 8/3 d" and activation memory during training.

⚡Key Problems Solved

plain ReLU/GELU MLPs have no input-conditional selection — information flows the same way for every token; the

4d

intermediate width inflates parameters and activations; SwiGLU lets different tokens traverse different "sub-networks" via input-dependent gating, improving quality at equal parameter count while actually shrinking parameters and training activation memory (