G6EasyPart G · Architectures & Parameter-Efficient Fine-Tuning

Vision Transformer (ViT) Patch Embedding

⚡Flashcard Drill➔

Divide 2D images into non-overlapping P x P patches flattened and projected into Transformer token embeddings.

⏱️ Time: O(B * H * W * C * D / P^2)
💾 Space: O(B * N * D)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def vit_patch_embedding_manual(
x: np.ndarray, # (B, C, H, W)
patch_size: int = 16,
d_model: int = 768,
projection_w: np.ndarray = None # (C * P * P, d_model)
) -> np.ndarray:
B, C, H, W = x.shape
P = patch_size
assert H % P == 0 and W % P == 0, "图片宽高必须能被 patch_size 整除"
num_patches_h = H // P
num_patches_w = W // P
num_patches =
# 1. 拆分切片: (B, C, num_h, P, num_w, P)
x = x.reshape(B, C, num_patches_h, P, num_patches_w, P)
# 置换轴: (B, num_h, num_w, P, P, C)
x = x.transpose(0, 2, 4, 3, 5, 1)
# 展平每个 patch: (B, num_patches, P * P * C)
patches = x.reshape(B, num_patches, P * P * C)
# 2. 线性投影到 d_model
if projection_w is None:
projection_w = np.random.randn(P * P * C, d_model) * 0.02
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does ViT handle inference at a higher image resolution than used during pretraining (e.g., 224x224 to 448x448)?
Next: Adam & AdamW Optimizer from Scratch →
💡

Core Mental Anchor / Mnemonic

Split HxW into P x P patches and project linearly (equivalent to Conv2d kernel=P, stride=P).

📐 Mathematical Derivation & Core Formula

X∈RH×W×C→Xp∈RN×(P2⋅C),N=HWP2X \in \mathbb{R}^{H \times W \times C} \to X_p \in \mathbb{R}^{N \times (P^2 \cdot C)}, \quad N = \frac{HW}{P^2}
### Tokenizing Images via Non-Overlapping Strided Convolution
Vision Transformer (ViT) splits an image X∈RB×C×H×WX \in \mathbb{R}^{B \times C \times H \times W} into N=HW/P2N = HW/P^2 patches of size P×PP \times P, flattens each to length P2CP^2 C, and linearly projects to dimension DD.
Algebraically, this patchify + projection step is identical to a single Conv2d layer with kernel_size=P, stride=P, and out_channels=D.

🔄 Tensor Dimensions & Shape Flow

(B, C, H, W) -> reshape & transpose patches -> (B, N, P*P*C) -> linear proj -> (B, N, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Prepend a learnable [CLS] token when doing classification ((B, N+1, D)).
  • Add positional embeddings before feeding patch tokens into Transformer encoder layers.

💻 Reference Code Implementation

import numpy as np

def vit_patch_embedding_manual(
    x: np.ndarray,          # (B, C, H, W)
    patch_size: int = 16,
    d_model: int = 768,
    projection_w: np.ndarray = None  # (C * P * P, d_model)
) -> np.ndarray:
    B, C, H, W = x.shape
    P = patch_size
    assert H % P == 0 and W % P == 0, "图片宽高必须能被 patch_size 整除"
    
    num_patches_h = H // P
    num_patches_w = W // P
    num_patches = num_patches_h * num_patches_w
    
    # 1. 拆分切片: (B, C, num_h, P, num_w, P)
    x = x.reshape(B, C, num_patches_h, P, num_patches_w, P)
    # 置换轴: (B, num_h, num_w, P, P, C)
    x = x.transpose(0, 2, 4, 3, 5, 1)
    # 展平每个 patch: (B, num_patches, P * P * C)
    patches = x.reshape(B, num_patches, P * P * C)
    
    # 2. 线性投影到 d_model
    if projection_w is None:
        projection_w = np.random.randn(P * P * C, d_model) * 0.02
        
    return patches @ projection_w  # (B, num_patches, d_model)

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
# 模拟 32x32 图像,patch_size=16,通道=3
x = np.random.randn(2, 3, 32, 32)
tokens = vit_patch_embedding_manual(x, patch_size=16, d_model=64)
# 32/16 = 2,共 2x2 = 4 个 patches
assert tokens.shape == (2, 4, 64), f"形状不匹配: {tokens.shape}"
print("✓ ViT Patch 切片嵌入自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How does ViT handle inference at a higher image resolution than used during pretraining (e.g., 224x224 to 448x448)?
Changing resolution changes the patch grid count. Learned 1D positional embeddings are reshaped to a 2D spatial grid (e.g., 14×1414 \times 14), resized via 2D bicubic interpolation to the target grid (28×2828 \times 28), and flattened back into tokens.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Pre-LN Transformer Residual Block AssemblyBack to Part G & All 105Next: Adam & AdamW Optimizer from Scratch →