M3-016M3: Deep Learning FoundationsActivation FunctionsMedium
⚡Flashcard Drill➔Mastery:

The Gated Linear Unit (GLU) Family and Why They Excel in Large Language Models

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M3-017 →
📐 Mathematical Definition
GLU(x)=(xW1)⊙σ(xW2)\mathrm{GLU}(x)=(xW_1)\odot\sigma(xW_2)
⚡ One-Sentence Core Answer
GLUs compute element-wise products between a linear projection and a non-linearly gated projection; variants like SwiGLU outperform standard activations by dynamically modulating feature flow.

📌 Key Takeaways

  • •
    GLU formulation: GLU(x)=(xW+b)⊗σ(xV+c)\text{GLU}(x) = (x W + b) \otimes \sigma(x V + c)
  • •
    SwiGLU variant: SwiGLU(x)=Swish1(xW)⊗(xV)\text{SwiGLU}(x) = \text{Swish}_1(x W) \otimes (x V); adopted by LLaMA, PaLM, and Mistral
  • •
    Parameter adjustment: because GLU requires 3 weight matrices instead of 2 in MLPs, hidden dimension is scaled by 2/32/3 to preserve parameter parity

📐 Mathematical Derivations

Mathematical Formulations (Dauphin et al., Shazeer 2020):
Standard MLP layer: FFN(x)=σ(xW1)W2\text{FFN}(x) = \sigma(x W_1) W_2.
Gated Linear Unit (GLU) replaces single non-linearity with a bilinear gating product:
- Bilinear: Bilinear(x)=(xW)⊗(xV)\text{Bilinear}(x) = (x W) \otimes (x V)
- GLU: GLU(x)=(xW)⊗σ(xV)\text{GLU}(x) = (x W) \otimes \sigma(x V)
- GeGLU: GeGLU(x)=GELU(xW)⊗(xV)\text{GeGLU}(x) = \text{GELU}(x W) \otimes (x V)
- SwiGLU: SwiGLU(x)=Swish1(xW)⊗(xV)=(xW⋅11+e−xW)⊗(xV)\text{SwiGLU}(x) = \text{Swish}_1(x W) \otimes (x V) = \left(x W \cdot \frac{1}{1 + e^{-x W}}\right) \otimes (x V).
Full SwiGLU MLP Block:
FFNSwiGLU(x)=(SiLU(xWgate)⊙(xWup))Wdown\text{FFN}_{\text{SwiGLU}}(x) = \left( \text{SiLU}(x W_{\text{gate}}) \odot (x W_{\text{up}}) \right) W_{\text{down}}.
Why it works: The linear path xVx V provides an unobstructed multiplicative highway where gradients can flow directly without being squashed by activation derivatives, while the gating path dynamically controls information pass-through.

🏭 Production Trade-offs

Parameter parity adjustment: Standard MLP has 8d28 d^2 parameters (two matrices of size d×4dd \times 4d). SwiGLU has three matrices (Wgate,Wup,WdownW_{\text{gate}}, W_{\text{up}}, W_{\text{down}}). To keep parameter count and FLOPs identical, the intermediate hidden dimension is adjusted from 4d4d to 83d≈2.67d\frac{8}{3}d \approx 2.67d (e.g., LLaMA sets hidden dimension to 23×4d=2.67d\frac{2}{3} \times 4d = 2.67d rounded to a multiple of 256).
⚠️ Common Interview Pitfalls
  • ✕
    Adopting SwiGLU with standard 4d4d intermediate dimension without realizing parameter count and FLOPs have increased by 50%
  • ✕
    Confusing the roles of WgateW_{\text{gate}} and WupW_{\text{up}} in SwiGLU implementations
🎯 Interviewer Follow-up Inquiries
  • ?
    Why is the intermediate hidden dimension of SwiGLU set to ≈83d\approx \frac{8}{3}d instead of 4d4d in LLaMA?
  • ?
    What gradient flow advantages does the bilinear branch in SwiGLU offer over standard feedforward layers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM3-015: Why Modern Transformers Use GELU and SiLU Instead of ReLU📋 Back to BankNext →M3-017: How Activation Function Choice Impacts Numerical Stability and Training Speed