G4MediumPart G · Architectures & Parameter-Efficient Fine-Tuning

adaLN-Zero (Adaptive LayerNorm with Zero Init)

⚡Flashcard Drill➔

Modulate DiT LayerNorm scale/shift and residual gates from conditioning vectors with zero initialization.

⏱️ Time: O(B * C * 6D + B * S * D)
💾 Space: O(B * 6D) 条件调制参数
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
class AdaLNZeroBlock:
def __init__(self, d_model: int, cond_dim: int):
self.d_model = d_model
# MLP 输出 6 个与 d_model 相同维度的调制参数: [gamma1, beta1, alpha1, gamma2, beta2, alpha2]
self.mlp_w = np.zeros((cond_dim, 6 * d_model)) # 最后一层全零初始化
self.mlp_b = np.zeros(6 * d_model)
def forward(self, x: np.ndarray, cond: np.ndarray) -> np.ndarray:
"""
x: (B, S, D)
cond: (B, cond_dim) 时间步 t 与类别 c 的条件特征
"""
B, S, D = x.shape
# 1. 调制参数投影
mod_params =
chunks = np.split(mod_params, 6, axis=-1)
gamma1, beta1, alpha1, gamma2, beta2, alpha2 = [c[:, np.newaxis, :] for c in chunks]
# 2. 简易 LayerNorm 模拟
mean = x.mean(axis=-1, keepdims=True)
var = x.var(axis=-1, keepdims=True)
x_norm = (x - mean) / np.sqrt(var + 1e-5)
# 3. 调制与模拟残差 (初始阶段由于 alpha1=0,残差直接为 0)
attn_sim = (1.0 + gamma1) * x_norm + beta1 # 调制输入
x =
return x
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What compute and memory advantages does adaLN-Zero offer over Cross-Attention conditioning?
Next: Pre-LN Transformer Residual Block Assembly →
💡

Core Mental Anchor / Mnemonic

Project timestep/class embedding to 6 zero-init modulation vectors; scale as (1 + gamma) for smooth identity start.

📐 Mathematical Derivation & Core Formula

y=α⋅Attention((1+γ)⋅LN(x)+β)+x,(γ,β,α)∼MLP(c)y = \alpha \cdot \mathrm{Attention}\left((1 + \gamma) \cdot \mathrm{LN}(x) + \beta\right) + x, \quad (\gamma, \beta, \alpha) \sim \mathrm{MLP}(c)
### Adaptive Conditioning & Zero-Initialized Residual Gating
In Diffusion Transformers (DiT, Peebles & Xie, 2022), conditioning vectors cc (timestep + label) pass through an MLP producing 6 modulation parameters (γ1,β1,α1,γ2,β2,α2)(\gamma_1, \beta_1, \alpha_1, \gamma_2, \beta_2, \alpha_2) to scale/shift LayerNorm features and gate residual branches.
Initializing the final MLP layer to zero sets γ=β=α=0\gamma = \beta = \alpha = 0 at step 0, reducing each DiT block initially to an identity residual map for stable deep training.

🔄 Tensor Dimensions & Shape Flow

cond: (B, C) -> MLP -> 6 modulation tensors (B, 1, D) -> scale/shift LN & gate residual alpha -> out

🛡️ Numerical Stability & Pitfalls Checklist

  • Zero-initialize the conditioning MLP output layer so every block begins as an identity residual pass.
  • Apply scale as (1 + gamma) so gamma = 0 starts cleanly at unit scale 1.0.

💻 Reference Code Implementation

import numpy as np

class AdaLNZeroBlock:
    def __init__(self, d_model: int, cond_dim: int):
        self.d_model = d_model
        # MLP 输出 6 个与 d_model 相同维度的调制参数: [gamma1, beta1, alpha1, gamma2, beta2, alpha2]
        self.mlp_w = np.zeros((cond_dim, 6 * d_model)) # 最后一层全零初始化
        self.mlp_b = np.zeros(6 * d_model)
        
    def forward(self, x: np.ndarray, cond: np.ndarray) -> np.ndarray:
        """
        x: (B, S, D)
        cond: (B, cond_dim) 时间步 t 与类别 c 的条件特征
        """
        B, S, D = x.shape
        # 1. 调制参数投影
        mod_params = cond @ self.mlp_w + self.mlp_b # (B, 6*D)
        chunks = np.split(mod_params, 6, axis=-1)
        gamma1, beta1, alpha1, gamma2, beta2, alpha2 = [c[:, np.newaxis, :] for c in chunks]
        
        # 2. 简易 LayerNorm 模拟
        mean = x.mean(axis=-1, keepdims=True)
        var = x.var(axis=-1, keepdims=True)
        x_norm = (x - mean) / np.sqrt(var + 1e-5)
        
        # 3. 调制与模拟残差 (初始阶段由于 alpha1=0,残差直接为 0)
        attn_sim = (1.0 + gamma1) * x_norm + beta1 # 调制输入
        x = x + alpha1 * attn_sim                  # 门控残差
        return x

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
block = AdaLNZeroBlock(d_model=8, cond_dim=4)
x = np.random.randn(2, 5, 8)
c = np.random.randn(2, 4)
out = block.forward(x, c)
# 全零初始化下,输出应严格等于输入原值!
assert np.allclose(out, x), "零初始化未达成恒等映射!"
print("✓ adaLN-Zero 扩散条件注入自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What compute and memory advantages does adaLN-Zero offer over Cross-Attention conditioning?
Cross-Attention adds pairwise quadratic attention maps over sequence tokens and conditioning tokens, whereas adaLN-Zero performs lightweight channel-wise elementwise affine scaling and gating without attention-matrix overhead.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: MoE Top-K Routing & Auxiliary Load Balancing LossBack to Part G & All 105Next: Pre-LN Transformer Residual Block Assembly →