M4-033M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Hard
⚡Flashcard Drill➔Mastery:
2D and 3D Rotary Position Embeddings (RoPE) for Vision and Video Multimodal Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
2D/3D RoPE decomposes channel dimensions into spatial (height, width) and temporal (time) sub-bands, rotating each sub-band by its corresponding coordinate to preserve multi-dimensional geometry.
📌 Key Takeaways
- •2D RoPE (Vision): allocates dimensions to vertical coordinate and dimensions to horizontal coordinate
- •3D RoPE (Video): splits channels into temporal , vertical , and horizontal sub-bands:
- •Multimodal integration: allows Vision Transformers and Video LLMs (Qwen2-VL) to process variable aspect ratios and framerates seamlessly
📐 Mathematical Derivations
Mathematical Formulations (Qwen2-VL, 2024; Su et al.):
Standard 1D RoPE flattens 2D images or 3D video patches into a 1D sequence, destroying 2D spatial adjacency and temporal periodicity.
① 2D RoPE for Images:
Let patch coordinates be with head dimension .
Split head vector into two halves: , where .
Apply 1D RoPE to each half independently using its spatial coordinate:
.
The dot product decomposes into spatial relative distances:
.
Attention scores depend strictly on 2D relative displacement , regardless of image aspect ratio or resolution.
② 3D RoPE for Video & Spatio-Temporal Data:
Partition head dimension into three segments: (e.g., for ).
.
Standard 1D RoPE flattens 2D images or 3D video patches into a 1D sequence, destroying 2D spatial adjacency and temporal periodicity.
① 2D RoPE for Images:
Let patch coordinates be with head dimension .
Split head vector into two halves: , where .
Apply 1D RoPE to each half independently using its spatial coordinate:
.
The dot product decomposes into spatial relative distances:
.
Attention scores depend strictly on 2D relative displacement , regardless of image aspect ratio or resolution.
② 3D RoPE for Video & Spatio-Temporal Data:
Partition head dimension into three segments: (e.g., for ).
.
🏭 Production Trade-offs
System design benefits: Native 2D/3D RoPE allows models like Qwen2-VL to handle arbitrary native image resolutions and variable video framerates without resizing images to static square grids, preserving high-resolution text and fine OCR details.
⚠️ Common Interview Pitfalls
- ✕Flattening 2D vision patches into 1D sequences and applying standard 1D RoPE, which introduces artificial line-break boundary jumps
- ✕Allocating unequal frequencies between and dimensions when spatial pixel resolution is isotropic
🎯 Interviewer Follow-up Inquiries
- ?How does 2D RoPE preserve spatial Euclidean distance invariance across image patches?
- ?Why is 3D RoPE essential for training unified Video-Language foundation models?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚