C2Medium★ Core Essential · Industrial BedrockPart C · Positional Encoding Evolution

Rotary Position Embedding (RoPE)

⚡Flashcard Drill➔

Apply 2D pairwise rotations to Q and K so dot products depend strictly on relative token distance.

⏱️ Time: O(B * H * S * D) 纯逐元素乘加
💾 Space: O(S * D) 只读旋转缓存
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def rotate_half(x: np.ndarray) -> np.ndarray:
"""
将最后一维两两配对交错取负:[-x2, x1, -x4, x3, ...]
常见工业实现:拆分为前半段与后半段 [-x2, x1]
"""
d = x.shape[-1]
x1 =
x2 = x[..., d // 2:]
return np.concatenate([-x2, x1], axis=-1)
def apply_rotary_pos_emb(x: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
"""
参数:
x: (B, H, S, D) Query 或 Key 张量
cos, sin: (1, 1, S, D) 预计算的旋转角度矩阵
返回:
旋转后的张量,形状恒为 (B, H, S, D)
"""
return (x * cos) + (rotate_half(x) * sin)
def precompute_rope_freqs(seq_len: int, dim: int, theta_base: float = 10000.0):
"""预计算 cos 与 sin 缓存"""
# 频率维度 half_dim
half_dim = dim // 2
freqs = 1.0 / (theta_base ** (np.arange(0, half_dim) / half_dim))
t = np.arange(seq_len)
# 外积得到角度网格 (S, half_dim)
angles = np.outer(t, freqs)
# 拼接为 (S, dim) 以匹配特征
angles = np.concatenate([angles, angles], axis=-1)
cos = np.cos(angles)[np.newaxis, np.newaxis, :, :] # (1, 1, S, D)
sin = np.sin(angles)[np.newaxis, np.newaxis, :, :]
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why is RoPE applied only to Query and Key vectors, and not to Value vectors?
Next: Attention with Linear Biases (ALiBi) →
💡

Core Mental Anchor / Mnemonic

Rotate Q/K pairs by m*theta: dot product depends strictly on relative offset (m - n).

📐 Mathematical Derivation & Core Formula

RΘ,md=diag(Rθ1,m,…,Rθd/2,m),⟨Rmq,Rnk⟩=g(q,k,m−n)R_{\Theta, m}^d = \mathrm{diag}\left(R_{\theta_1, m}, \dots, R_{\theta_{d/2}, m}\right), \quad \langle R_m q, R_n k \rangle = g(q, k, m - n)
### Complex Inner-Product Identity
RoPE (Su et al., 2021) finds transformations f(q,m)f(q, m) and f(k,n)f(k, n) whose inner product depends strictly on relative distance m−nm - n:
⟨f(q,m),f(k,n)⟩=g(q,k,m−n)\langle f(q, m), f(k, n) \rangle = g(q, k, m - n)
In the 2D complex plane, multiplying by eimθe^{i m \theta} rotates a vector by angle mθm\theta:
⟨qeimθ,keinθ⟩=Re[(qeimθ)(keinθ)∗]=Re[qk∗ei(m−n)θ]\langle q e^{i m \theta}, k e^{i n \theta} \rangle = \mathrm{Re}[(q e^{i m \theta})(k e^{i n \theta})^*] = \mathrm{Re}[q k^* e^{i(m - n)\theta}]
For DD-dimensional vectors, we apply 2D rotations independently to D/2D/2 pairs:
Rmx=x⊙cos⁡(mθ)+[−x2,x1,−x4,x3,… ]⊙sin⁡(mθ)R_m x = x \odot \cos(m\theta) + [-x_2, x_1, -x_4, x_3, \dots] \odot \sin(m\theta)

🔄 Tensor Dimensions & Shape Flow

x: (B, H, S, D) -> cos/sin: (1, 1, S, D) -> x * cos + rotate_half(x) * sin -> (B, H, S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Ensure Q and K use the exact same rotate_half pairing convention (interleaved vs. split-half).
  • Apply RoPE only to Queries and Keys—never rotate Value vectors.
  • Long-context variants raise theta base from 10,000 (e.g., 500,000 in LLaMA-3) to preserve phase resolution.

💻 Reference Code Implementation

import numpy as np

def rotate_half(x: np.ndarray) -> np.ndarray:
    """
    将最后一维两两配对交错取负:[-x2, x1, -x4, x3, ...]
    常见工业实现:拆分为前半段与后半段 [-x2, x1]
    """
    d = x.shape[-1]
    x1 = x[..., :d // 2]
    x2 = x[..., d // 2:]
    return np.concatenate([-x2, x1], axis=-1)

def apply_rotary_pos_emb(x: np.ndarray, cos: np.ndarray, sin: np.ndarray) -> np.ndarray:
    """
    参数:
        x: (B, H, S, D) Query 或 Key 张量
        cos, sin: (1, 1, S, D) 预计算的旋转角度矩阵
    返回:
        旋转后的张量,形状恒为 (B, H, S, D)
    """
    return (x * cos) + (rotate_half(x) * sin)

def precompute_rope_freqs(seq_len: int, dim: int, theta_base: float = 10000.0):
    """预计算 cos 与 sin 缓存"""
    # 频率维度 half_dim
    half_dim = dim // 2
    freqs = 1.0 / (theta_base ** (np.arange(0, half_dim) / half_dim))
    t = np.arange(seq_len)
    # 外积得到角度网格 (S, half_dim)
    angles = np.outer(t, freqs)
    # 拼接为 (S, dim) 以匹配特征
    angles = np.concatenate([angles, angles], axis=-1)
    cos = np.cos(angles)[np.newaxis, np.newaxis, :, :]  # (1, 1, S, D)
    sin = np.sin(angles)[np.newaxis, np.newaxis, :, :]
    return cos, sin

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
B, H, S, D = 1, 2, 4, 8
q = np.random.randn(B, H, S, D)
k = np.random.randn(B, H, S, D)
cos, sin = precompute_rope_freqs(seq_len=S, dim=D)
q_rot = apply_rotary_pos_emb(q, cos, sin)
k_rot = apply_rotary_pos_emb(k, cos, sin)
assert q_rot.shape == (B, H, S, D)
assert k_rot.shape == (B, H, S, D)
print("✓ RoPE 旋转位置编码自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why is RoPE applied only to Query and Key vectors, and not to Value vectors?
Queries and Keys compute pairwise alignment scores via dot products, where relative rotation angle m−nm - n encodes relative distance. Values carry content summed by attention weights; rotating Values would mix different phases across positions when aggregated.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Sinusoidal Positional EncodingBack to Part C & All 105Next: Attention with Linear Biases (ALiBi) →