A4Medium★ Core Essential · Industrial BedrockPart A · Core Kernels & Activation Functions
SwiGLU Gated Feed-Forward Network
Flagship FFN operator powering LLaMA and DeepSeek, combining SiLU gated branch with up-projection and down-projection.
⏱️ Time Complexity:
O(3 * B * S * D * D_ffn)💾 Space Complexity:
O(B * S * D_ffn)💡
Core Mental Anchor / Mnemonic
Gate and up run in parallel; SiLU modulates via Hadamard product; down-projection restores dimension
📐 Mathematical Derivation & Core Formula
### Architectural Evolution & Math
Proposed by Noam Shazeer (2020), SwiGLU replaces the intermediate FFN with a bilinear gated unit:
To preserve parameter count compared to a standard FFN, is typically calibrated to (e.g. 11008 in LLaMA-7B).
Proposed by Noam Shazeer (2020), SwiGLU replaces the intermediate FFN with a bilinear gated unit:
To preserve parameter count compared to a standard FFN, is typically calibrated to (e.g. 11008 in LLaMA-7B).
🔄 Tensor Dimensions & Shape Flow
(B, S, D) -> parallel projections -> 2x (B, S, D_ffn) -> SiLU(gate) * up -> (B, S, D_ffn) -> down-projection -> (B, S, D)
🛡️ Industrial Numerical Stability & Pitfalls
- SiLU requires piecewise exp protection for extreme negative logits
- Fuse W_gate and W_up into [W_gate, W_up] (D, 2*D_ffn) for single GEMM invocation
- Ensure D_ffn aligns to multiples of 128 or 256 for Tensor Core memory layout
💻 Industrial Code Implementation
import numpy as np
def swiglu(x: np.ndarray, W_gate: np.ndarray, W_up: np.ndarray, W_down: np.ndarray) -> np.ndarray:
"""
SwiGLU Feed-Forward Network operator.
Dimensions:
x: (B, S, D)
W_gate, W_up: (D, D_ffn)
W_down: (D_ffn, D)
"""
gate = x @ W_gate # (B, S, D_ffn)
pos = gate >= 0
sig = np.empty_like(gate)
sig[pos] = 1.0 / (1.0 + np.exp(-gate[pos]))
ez = np.exp(gate[~pos])
sig[~pos] = ez / (1.0 + ez)
silu_gate = gate * sig # SiLU(gate)
up = x @ W_up # (B, S, D_ffn)
hidden = silu_gate * up # Hadamard product
return hidden @ W_down # (B, S, D)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter to verify correctness:
import numpy as np
B, S, D, D_ffn = 2, 4, 8, 16
x = np.random.randn(B, S, D)
Wg = np.random.randn(D, D_ffn)
Wu = np.random.randn(D, D_ffn)
Wd = np.random.randn(D_ffn, D)
res = swiglu(x, Wg, Wu, Wd)
assert res.shape == (B, S, D)
assert not np.isnan(res).any()
print("✓ SwiGLU assertion passed")🎯 Core Architecture Follow-up Q&A
Q1:How to split SwiGLU in Tensor Parallelism (Megatron-LM style)?
W_gate and W_up are Column-Parallel. Each GPU computes its local chunk of channels and multiplies them locally. W_down is Row-Parallel, followed by a single All-Reduce across GPUs.