G6EasyPart G · Architectures & Parameter-Efficient Fine-Tuning
Vision Transformer (ViT) Patch Embedding
Divide 2D images into non-overlapping P x P patches flattened and projected into Transformer token embeddings.
⏱️ Time:
O(B * H * W * C * D / P^2)💾 Space:
O(B * N * D)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def vit_patch_embedding_manual(
x: np.ndarray, # (B, C, H, W)
patch_size: int = 16,
d_model: int = 768,
projection_w: np.ndarray = None # (C * P * P, d_model)
) -> np.ndarray:
B, C, H, W = x.shape
P = patch_size
assert H % P == 0 and W % P == 0, "图片宽高必须能被 patch_size 整除"
num_patches_h = H // P
num_patches_w = W // P
num_patches =
# 1. 拆分切片: (B, C, num_h, P, num_w, P)
x = x.reshape(B, C, num_patches_h, P, num_patches_w, P)
# 置换轴: (B, num_h, num_w, P, P, C)
x = x.transpose(0, 2, 4, 3, 5, 1)
# 展平每个 patch: (B, num_patches, P * P * C)
patches = x.reshape(B, num_patches, P * P * C)
# 2. 线性投影到 d_model
if projection_w is None:
projection_w = np.random.randn(P * P * C, d_model) * 0.02
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How does ViT handle inference at a higher image resolution than used during pretraining (e.g., 224x224 to 448x448)?
💡
Core Mental Anchor / Mnemonic
Split HxW into P x P patches and project linearly (equivalent to Conv2d kernel=P, stride=P).
📐 Mathematical Derivation & Core Formula
### Tokenizing Images via Non-Overlapping Strided Convolution
Vision Transformer (ViT) splits an image into patches of size , flattens each to length , and linearly projects to dimension .
Algebraically, this patchify + projection step is identical to a single
Vision Transformer (ViT) splits an image into patches of size , flattens each to length , and linearly projects to dimension .
Algebraically, this patchify + projection step is identical to a single
Conv2d layer with kernel_size=P, stride=P, and out_channels=D.🔄 Tensor Dimensions & Shape Flow
(B, C, H, W) -> reshape & transpose patches -> (B, N, P*P*C) -> linear proj -> (B, N, D)
🛡️ Numerical Stability & Pitfalls Checklist
- Prepend a learnable
[CLS]token when doing classification ((B, N+1, D)). - Add positional embeddings before feeding patch tokens into Transformer encoder layers.
💻 Reference Code Implementation
import numpy as np
def vit_patch_embedding_manual(
x: np.ndarray, # (B, C, H, W)
patch_size: int = 16,
d_model: int = 768,
projection_w: np.ndarray = None # (C * P * P, d_model)
) -> np.ndarray:
B, C, H, W = x.shape
P = patch_size
assert H % P == 0 and W % P == 0, "图片宽高必须能被 patch_size 整除"
num_patches_h = H // P
num_patches_w = W // P
num_patches = num_patches_h * num_patches_w
# 1. 拆分切片: (B, C, num_h, P, num_w, P)
x = x.reshape(B, C, num_patches_h, P, num_patches_w, P)
# 置换轴: (B, num_h, num_w, P, P, C)
x = x.transpose(0, 2, 4, 3, 5, 1)
# 展平每个 patch: (B, num_patches, P * P * C)
patches = x.reshape(B, num_patches, P * P * C)
# 2. 线性投影到 d_model
if projection_w is None:
projection_w = np.random.randn(P * P * C, d_model) * 0.02
return patches @ projection_w # (B, num_patches, d_model)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
# 模拟 32x32 图像,patch_size=16,通道=3
x = np.random.randn(2, 3, 32, 32)
tokens = vit_patch_embedding_manual(x, patch_size=16, d_model=64)
# 32/16 = 2,共 2x2 = 4 个 patches
assert tokens.shape == (2, 4, 64), f"形状不匹配: {tokens.shape}"
print("✓ ViT Patch 切片嵌入自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How does ViT handle inference at a higher image resolution than used during pretraining (e.g., 224x224 to 448x448)?
Changing resolution changes the patch grid count. Learned 1D positional embeddings are reshaped to a 2D spatial grid (e.g., ), resized via 2D bicubic interpolation to the target grid (), and flattened back into tokens.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.