G5Medium★ Core Essential · Industrial BedrockPart G · Architectures & Parameter-Efficient Fine-Tuning

Pre-LN Transformer Residual Block Assembly

⚡Flashcard Drill➔

Place normalization before Attention and FFN inside residual branches to preserve a clean residual gradient path.

⏱️ Time: O(B * S^2 * D + B * S * D^2)
💾 Space: O(B * S * D)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def simple_layer_norm(x: np.ndarray, eps: float = 1e-5) -> np.ndarray:
mean =
var = np.var(x, axis=-1, keepdims=True)
return (x - mean) / np.sqrt(var + eps)
class PreLNTransformerBlock:
def __init__(self, d_model: int):
self.d_model = d_model
# 简化版自注意力与 MLP 投影矩阵
self.W_attn = np.random.randn(d_model, d_model) * 0.02
self.W_mlp1 = np.random.randn(d_model, 4 * d_model) * 0.02
self.W_mlp2 = np.random.randn(4 * d_model, d_model) * 0.02
def forward(self, x: np.ndarray) -> np.ndarray:
"""
x: (B, S, D)
"""
# 1. 第一个 Pre-LN 子层: Attention
norm_x1 = simple_layer_norm(x)
# 模拟注意力计算
attn_out = norm_x1 @ self.W_attn
# 残差相加
x = x + attn_out
# 2. 第二个 Pre-LN 子层: MLP
norm_x2 = simple_layer_norm(x)
# GELU / ReLU 模拟
hidden = np.maximum(0, norm_x2 @ self.W_mlp1)
mlp_out = hidden @ self.W_mlp2
# 残差相加
x =
return x
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How do LLaMA and DeepSeek combine RMSNorm with Pre-LN?
Next: Vision Transformer (ViT) Patch Embedding →
💡

Core Mental Anchor / Mnemonic

Normalize inside sub-branches before Attn/MLP and leave the residual highway untouched.

📐 Mathematical Derivation & Core Formula

x=x+MHA(LN(x)),x=x+MLP(LN(x))x = x + \mathrm{MHA}(\mathrm{LN}(x)), \quad x = x + \mathrm{MLP}(\mathrm{LN}(x))
### Residual Stream & Unobstructed Gradient Highway
Post-LN (x=LN(x+SubLayer(x))x = \mathrm{LN}(x + \mathrm{SubLayer}(x))) applies normalization along the main trunk, attenuating early-layer gradients and requiring careful warmup.
Pre-LN moves normalization inside each residual branch:
xl+1=xl+SubLayer(LN(xl))  ⟹  xL=x0+∑l=0L−1SubLayer(LN(xl))x_{l+1} = x_l + \mathrm{SubLayer}(\mathrm{LN}(x_l)) \implies x_L = x_0 + \sum_{l=0}^{L-1} \mathrm{SubLayer}(\mathrm{LN}(x_l))
This preserves an identity gradient path from output to input layers, stabilizing deep Transformer optimization.

🔄 Tensor Dimensions & Shape Flow

x: (B, S, D) -> LN -> Attn -> + x -> LN -> MLP -> + x -> out: (B, S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Always apply a final normalization layer (norm_f) after the last Pre-LN block before the LM head.
  • Keep the residual trunk un-normalized across layers so early-layer gradients propagate cleanly.

💻 Reference Code Implementation

import numpy as np

def simple_layer_norm(x: np.ndarray, eps: float = 1e-5) -> np.ndarray:
    mean = np.mean(x, axis=-1, keepdims=True)
    var = np.var(x, axis=-1, keepdims=True)
    return (x - mean) / np.sqrt(var + eps)

class PreLNTransformerBlock:
    def __init__(self, d_model: int):
        self.d_model = d_model
        # 简化版自注意力与 MLP 投影矩阵
        self.W_attn = np.random.randn(d_model, d_model) * 0.02
        self.W_mlp1 = np.random.randn(d_model, 4 * d_model) * 0.02
        self.W_mlp2 = np.random.randn(4 * d_model, d_model) * 0.02
        
    def forward(self, x: np.ndarray) -> np.ndarray:
        """
        x: (B, S, D)
        """
        # 1. 第一个 Pre-LN 子层: Attention
        norm_x1 = simple_layer_norm(x)
        # 模拟注意力计算
        attn_out = norm_x1 @ self.W_attn
        # 残差相加
        x = x + attn_out
        
        # 2. 第二个 Pre-LN 子层: MLP
        norm_x2 = simple_layer_norm(x)
        # GELU / ReLU 模拟
        hidden = np.maximum(0, norm_x2 @ self.W_mlp1)
        mlp_out = hidden @ self.W_mlp2
        # 残差相加
        x = x + mlp_out
        
        return x

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
block = PreLNTransformerBlock(d_model=16)
x = np.random.randn(2, 4, 16)
out = block.forward(x)
assert out.shape == (2, 4, 16)
assert not np.isnan(out).any()
print("✓ Pre-LN Transformer 残差块装配自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How do LLaMA and DeepSeek combine RMSNorm with Pre-LN?
They replace standard LayerNorm with RMSNorm before each Attention/FFN branch and place a final RMSNorm after the last block, combining Pre-LN gradient stability with lower memory-access overhead.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: adaLN-Zero (Adaptive LayerNorm with Zero Init)Back to Part G & All 105Next: Vision Transformer (ViT) Patch Embedding →