C2Medium★ Core Essential · Industrial BedrockPart C · Positional Encoding Evolution
Rotary Position Embedding (RoPE)
Apply 2D pairwise rotations to Q and K so dot products depend strictly on relative token distance.
⏱️ Time:
O(B * H * S * D) 纯逐元素乘加💾 Space:
O(S * D) 只读旋转缓存🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def rotate_half(x: np.ndarray) -> np.ndarray:
"""
将最后一维两两配对交错取负:[-x2, x1, -x4, x3, ...]
常见工业实现:拆分为前半段与后半段 [-x2, x1]
"""
d = x.shape[-1]
x1 =
x2 = x[..., d // 2:]
return np.concatenate([-x2, x1], axis=-1)
def apply_rotary_pos_emb(x: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
"""
参数:
x: (B, H, S, D) Query 或 Key 张量
cos, sin: (1, 1, S, D) 预计算的旋转角度矩阵
返回:
旋转后的张量,形状恒为 (B, H, S, D)
"""
return (x * cos) + (rotate_half(x) * sin)
def precompute_rope_freqs(seq_len: int, dim: int, theta_base: float = 10000.0):
"""预计算 cos 与 sin 缓存"""
# 频率维度 half_dim
half_dim = dim // 2
freqs = 1.0 / (theta_base ** (np.arange(0, half_dim) / half_dim))
t = np.arange(seq_len)
# 外积得到角度网格 (S, half_dim)
angles = np.outer(t, freqs)
# 拼接为 (S, dim) 以匹配特征
angles = np.concatenate([angles, angles], axis=-1)
cos = np.cos(angles)[np.newaxis, np.newaxis, :, :] # (1, 1, S, D)
sin = np.sin(angles)[np.newaxis, np.newaxis, :, :]
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is RoPE applied only to Query and Key vectors, and not to Value vectors?
💡
Core Mental Anchor / Mnemonic
Rotate Q/K pairs by m*theta: dot product depends strictly on relative offset (m - n).
📐 Mathematical Derivation & Core Formula
### Complex Inner-Product Identity
RoPE (Su et al., 2021) finds transformations and whose inner product depends strictly on relative distance :
In the 2D complex plane, multiplying by rotates a vector by angle :
For -dimensional vectors, we apply 2D rotations independently to pairs:
RoPE (Su et al., 2021) finds transformations and whose inner product depends strictly on relative distance :
In the 2D complex plane, multiplying by rotates a vector by angle :
For -dimensional vectors, we apply 2D rotations independently to pairs:
🔄 Tensor Dimensions & Shape Flow
x: (B, H, S, D) -> cos/sin: (1, 1, S, D) -> x * cos + rotate_half(x) * sin -> (B, H, S, D)
🛡️ Numerical Stability & Pitfalls Checklist
- Ensure Q and K use the exact same rotate_half pairing convention (interleaved vs. split-half).
- Apply RoPE only to Queries and Keys—never rotate Value vectors.
- Long-context variants raise theta base from 10,000 (e.g., 500,000 in LLaMA-3) to preserve phase resolution.
💻 Reference Code Implementation
import numpy as np
def rotate_half(x: np.ndarray) -> np.ndarray:
"""
将最后一维两两配对交错取负:[-x2, x1, -x4, x3, ...]
常见工业实现:拆分为前半段与后半段 [-x2, x1]
"""
d = x.shape[-1]
x1 = x[..., :d // 2]
x2 = x[..., d // 2:]
return np.concatenate([-x2, x1], axis=-1)
def apply_rotary_pos_emb(x: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
"""
参数:
x: (B, H, S, D) Query 或 Key 张量
cos, sin: (1, 1, S, D) 预计算的旋转角度矩阵
返回:
旋转后的张量,形状恒为 (B, H, S, D)
"""
return (x * cos) + (rotate_half(x) * sin)
def precompute_rope_freqs(seq_len: int, dim: int, theta_base: float = 10000.0):
"""预计算 cos 与 sin 缓存"""
# 频率维度 half_dim
half_dim = dim // 2
freqs = 1.0 / (theta_base ** (np.arange(0, half_dim) / half_dim))
t = np.arange(seq_len)
# 外积得到角度网格 (S, half_dim)
angles = np.outer(t, freqs)
# 拼接为 (S, dim) 以匹配特征
angles = np.concatenate([angles, angles], axis=-1)
cos = np.cos(angles)[np.newaxis, np.newaxis, :, :] # (1, 1, S, D)
sin = np.sin(angles)[np.newaxis, np.newaxis, :, :]
return cos, sin
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
B, H, S, D = 1, 2, 4, 8
q = np.random.randn(B, H, S, D)
k = np.random.randn(B, H, S, D)
cos, sin = precompute_rope_freqs(seq_len=S, dim=D)
q_rot = apply_rotary_pos_emb(q, cos, sin)
k_rot = apply_rotary_pos_emb(k, cos, sin)
assert q_rot.shape == (B, H, S, D)
assert k_rot.shape == (B, H, S, D)
print("✓ RoPE 旋转位置编码自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why is RoPE applied only to Query and Key vectors, and not to Value vectors?
Queries and Keys compute pairwise alignment scores via dot products, where relative rotation angle encodes relative distance. Values carry content summed by attention weights; rotating Values would mix different phases across positions when aggregated.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.