M4-033M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Hard
⚡Flashcard Drill➔Mastery:

2D and 3D Rotary Position Embeddings (RoPE) for Vision and Video Multimodal Models

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-034 →
📐 Mathematical Definition
M-RoPE: split d into (dt,dh,dw) segments, rotate each by its own coordinate\text{M-RoPE}:\ \text{split }d\ \text{into }(d_t,d_h,d_w)\ \text{segments, rotate each by its own coordinate}
⚡ One-Sentence Core Answer
2D/3D RoPE decomposes channel dimensions into spatial (height, width) and temporal (time) sub-bands, rotating each sub-band by its corresponding coordinate to preserve multi-dimensional geometry.

📌 Key Takeaways

  • •
    2D RoPE (Vision): allocates d/2d/2 dimensions to vertical coordinate yy and d/2d/2 dimensions to horizontal coordinate xx
  • •
    3D RoPE (Video): splits channels into temporal tt, vertical yy, and horizontal xx sub-bands: RoPE3D=[R(t),R(y),R(x)]\text{RoPE}_{3D} = [R(t), R(y), R(x)]
  • •
    Multimodal integration: allows Vision Transformers and Video LLMs (Qwen2-VL) to process variable aspect ratios and framerates seamlessly

📐 Mathematical Derivations

Mathematical Formulations (Qwen2-VL, 2024; Su et al.):
Standard 1D RoPE flattens 2D images or 3D video patches into a 1D sequence, destroying 2D spatial adjacency and temporal periodicity.
① 2D RoPE for Images:
Let patch coordinates be (y,x)∈R2(y, x) \in \mathbb{R}^2 with head dimension dd.
Split head vector into two halves: q=[q(y),q(x)]q = [q^{(y)}, q^{(x)}], where q(y),q(x)∈Rd/2q^{(y)}, q^{(x)} \in \mathbb{R}^{d/2}.
Apply 1D RoPE to each half independently using its spatial coordinate:
q~=[R(y;Θ(y))q(y),  R(x;Θ(x))q(x)]\tilde{q} = \left[ R(y; \Theta^{(y)}) q^{(y)}, \; R(x; \Theta^{(x)}) q^{(x)} \right].
The dot product decomposes into spatial relative distances:
q~Tk~=⟨q(y),R(yq−yk)k(y)⟩+⟨q(x),R(xq−xk)k(x)⟩\tilde{q}^T \tilde{k} = \langle q^{(y)}, R(y_q - y_k) k^{(y)} \rangle + \langle q^{(x)}, R(x_q - x_k) k^{(x)} \rangle.
Attention scores depend strictly on 2D relative displacement (Δy,Δx)(\Delta y, \Delta x), regardless of image aspect ratio or resolution.
② 3D RoPE for Video & Spatio-Temporal Data:
Partition head dimension into three segments: d=dt+dy+dxd = d_t + d_y + d_x (e.g., dt=32,dy=48,dx=48d_t = 32, d_y = 48, d_x = 48 for d=128d=128).
q~=[R(t;Θ(t))q(t),  R(y;Θ(y))q(y),  R(x;Θ(x))q(x)]\tilde{q} = \left[ R(t; \Theta^{(t)}) q^{(t)}, \; R(y; \Theta^{(y)}) q^{(y)}, \; R(x; \Theta^{(x)}) q^{(x)} \right].

🏭 Production Trade-offs

System design benefits: Native 2D/3D RoPE allows models like Qwen2-VL to handle arbitrary native image resolutions and variable video framerates without resizing images to static square grids, preserving high-resolution text and fine OCR details.
⚠️ Common Interview Pitfalls
  • ✕
    Flattening 2D vision patches into 1D sequences and applying standard 1D RoPE, which introduces artificial line-break boundary jumps
  • ✕
    Allocating unequal frequencies between xx and yy dimensions when spatial pixel resolution is isotropic
🎯 Interviewer Follow-up Inquiries
  • ?
    How does 2D RoPE preserve spatial Euclidean distance invariance across image patches?
  • ?
    Why is 3D RoPE essential for training unified Video-Language foundation models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-032: ALiBi (Attention with Linear Biases): Mechanisms and Zero-Shot Length Extrapolation📋 Back to BankNext →M4-034: Benchmarking Positional Encodings and Context Length Extrapolation