A4Medium★ Core Essential · Industrial BedrockPart A · Core Kernels & Activation Functions

SwiGLU Gated Feed-Forward Network

⚡Flashcard Drill➔

Flagship FFN operator powering LLaMA and DeepSeek, combining SiLU gated branch with up-projection and down-projection.

⏱️ Time: O(3 * B * S * D * D_ffn)
💾 Space: O(B * S * D_ffn)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def swiglu(x: np.ndarray, W_gate: np.ndarray, W_up: np.ndarray, W_down: np.ndarray) -> np.ndarray:
"""
SwiGLU Feed-Forward Network operator.
Dimensions:
x: (B, S, D)
W_gate, W_up: (D, D_ffn)
W_down: (D_ffn, D)
"""
gate =
pos = gate >= 0
sig = np.empty_like(gate)
sig[pos] = 1.0 / (1.0 + np.exp(-gate[pos]))
ez = np.exp(gate[~pos])
sig[~pos] = ez / (1.0 + ez)
silu_gate = gate * sig # SiLU(gate)
up = x @ W_up # (B, S, D_ffn)
hidden = silu_gate * up # Hadamard product
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How to split SwiGLU in Tensor Parallelism (Megatron-LM style)?
Next: Softplus & LeakyReLU with Overflow Guards →
💡

Core Mental Anchor / Mnemonic

Gate and up run in parallel; SiLU modulates via Hadamard product; down-projection restores dimension

📐 Mathematical Derivation & Core Formula

SwiGLU(x)=(SiLU(xWgate)⊙xWup)Wdown\mathrm{SwiGLU}(x) = \left(\mathrm{SiLU}(x W_{\mathrm{gate}}) \odot x W_{\mathrm{up}}\right) W_{\mathrm{down}}
### Architectural Evolution & Math
Proposed by Noam Shazeer (2020), SwiGLU replaces the intermediate FFN with a bilinear gated unit:
SwiGLU(x)=(SiLU(xWgate)⊙xWup)Wdown\mathrm{SwiGLU}(x) = (\mathrm{SiLU}(x W_{\mathrm{gate}}) \odot x W_{\mathrm{up}}) W_{\mathrm{down}}
To preserve parameter count compared to a standard 4d4d FFN, DffnD_{\mathrm{ffn}} is typically calibrated to ≈83d\approx \frac{8}{3} d (e.g. 11008 in LLaMA-7B).

🔄 Tensor Dimensions & Shape Flow

(B, S, D) -> parallel projections -> 2x (B, S, D_ffn) -> SiLU(gate) * up -> (B, S, D_ffn) -> down-projection -> (B, S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • SiLU requires piecewise exp protection for extreme negative logits
  • Fuse W_gate and W_up into [W_gate, W_up] (D, 2*D_ffn) for single GEMM invocation
  • Ensure D_ffn aligns to multiples of 128 or 256 for Tensor Core memory layout

💻 Reference Code Implementation

import numpy as np

def swiglu(x: np.ndarray, W_gate: np.ndarray, W_up: np.ndarray, W_down: np.ndarray) -> np.ndarray:
    """
    SwiGLU Feed-Forward Network operator.
    Dimensions:
        x: (B, S, D)
        W_gate, W_up: (D, D_ffn)
        W_down: (D_ffn, D)
    """
    gate = x @ W_gate                      # (B, S, D_ffn)
    pos = gate >= 0
    sig = np.empty_like(gate)
    sig[pos] = 1.0 / (1.0 + np.exp(-gate[pos]))
    ez = np.exp(gate[~pos])
    sig[~pos] = ez / (1.0 + ez)
    silu_gate = gate * sig                 # SiLU(gate)
    
    up = x @ W_up                          # (B, S, D_ffn)
    hidden = silu_gate * up                # Hadamard product
    return hidden @ W_down                 # (B, S, D)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, S, D, D_ffn = 2, 4, 8, 16
x = np.random.randn(B, S, D)
Wg = np.random.randn(D, D_ffn)
Wu = np.random.randn(D, D_ffn)
Wd = np.random.randn(D_ffn, D)
res = swiglu(x, Wg, Wu, Wd)
assert res.shape == (B, S, D)
assert not np.isnan(res).any()
print("✓ SwiGLU assertion passed")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How to split SwiGLU in Tensor Parallelism (Megatron-LM style)?
W_gate and W_up are Column-Parallel. Each GPU computes its local chunk of channels and multiplies them locally. W_down is Row-Parallel, followed by a single All-Reduce across GPUs.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Gaussian Error Linear Unit (GELU)Back to Part A & All 105Next: Softplus & LeakyReLU with Overflow Guards →