M3-016M3: Deep Learning FoundationsActivation FunctionsMedium
⚡Flashcard Drill➔Mastery:
The Gated Linear Unit (GLU) Family and Why They Excel in Large Language Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
GLUs compute element-wise products between a linear projection and a non-linearly gated projection; variants like SwiGLU outperform standard activations by dynamically modulating feature flow.
📌 Key Takeaways
- •GLU formulation:
- •SwiGLU variant: ; adopted by LLaMA, PaLM, and Mistral
- •Parameter adjustment: because GLU requires 3 weight matrices instead of 2 in MLPs, hidden dimension is scaled by to preserve parameter parity
📐 Mathematical Derivations
Mathematical Formulations (Dauphin et al., Shazeer 2020):
Standard MLP layer: .
Gated Linear Unit (GLU) replaces single non-linearity with a bilinear gating product:
- Bilinear:
- GLU:
- GeGLU:
- SwiGLU: .
Full SwiGLU MLP Block:
.
Why it works: The linear path provides an unobstructed multiplicative highway where gradients can flow directly without being squashed by activation derivatives, while the gating path dynamically controls information pass-through.
Standard MLP layer: .
Gated Linear Unit (GLU) replaces single non-linearity with a bilinear gating product:
- Bilinear:
- GLU:
- GeGLU:
- SwiGLU: .
Full SwiGLU MLP Block:
.
Why it works: The linear path provides an unobstructed multiplicative highway where gradients can flow directly without being squashed by activation derivatives, while the gating path dynamically controls information pass-through.
🏭 Production Trade-offs
Parameter parity adjustment: Standard MLP has parameters (two matrices of size ). SwiGLU has three matrices (). To keep parameter count and FLOPs identical, the intermediate hidden dimension is adjusted from to (e.g., LLaMA sets hidden dimension to rounded to a multiple of 256).
⚠️ Common Interview Pitfalls
- ✕Adopting SwiGLU with standard intermediate dimension without realizing parameter count and FLOPs have increased by 50%
- ✕Confusing the roles of and in SwiGLU implementations
🎯 Interviewer Follow-up Inquiries
- ?Why is the intermediate hidden dimension of SwiGLU set to instead of in LLaMA?
- ?What gradient flow advantages does the bilinear branch in SwiGLU offer over standard feedforward layers?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚