⚡ LLMsID: positional-encoding

Positional Encoding RoPE/ALiBi

Q: RoPE rotation formula? Why does the inner product depend only on the relative offset $m - n$?

【Positional Encoding RoPE/ALiBi 核心考点 1】RoPE rotation formula? Why does the inner product depend only on the relative offset $m - n$?。技术解析：Definition: RoPE (Rotary Position Embedding) rotates the 2j-th and 2j+1-th dims of Q/K by angle m \theta_j with \theta_j = 10000^{-2j/d}, so the inner product of any positions m, n depends only on their relative distance m - n; ALiBi instead adds a linear bias -m \cdot |i - j| to... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/positional-encoding

Q: RoPE wavelength: period of the lowest-frequency dim? When does base=10000 cause periodic ambiguity (~62K tokens; LLaMA-3 uses 500000)?

【Positional Encoding RoPE/ALiBi 核心考点 2】RoPE wavelength: period of the lowest-frequency dim? When does base=10000 cause periodic ambiguity (~62K tokens; LLaMA-3 uses 500000)?。技术解析：Definition: RoPE (Rotary Position Embedding) rotates the 2j-th and 2j+1-th dims of Q/K by angle m \theta_j with \theta_j = 10000^{-2j/d}, so the inner product of any positions m, n depends only on their relative distance m - n; ALiBi instead adds a linear bias -m \cdot |i - j| to... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/positional-encoding

Q: RoPE vs absolute and learnable position embeddings?

【Positional Encoding RoPE/ALiBi 核心考点 3】RoPE vs absolute and learnable position embeddings?。技术解析：Definition: RoPE (Rotary Position Embedding) rotates the 2j-th and 2j+1-th dims of Q/K by angle m \theta_j with \theta_j = 10000^{-2j/d}, so the inner product of any positions m, n depends only on their relative distance m - n; ALiBi instead adds a linear bias -m \cdot |i - j| to... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/positional-encoding

Q: How ALiBi works, and its pros and cons?

【Positional Encoding RoPE/ALiBi 核心考点 4】How ALiBi works, and its pros and cons?。技术解析：Definition: RoPE (Rotary Position Embedding) rotates the 2j-th and 2j+1-th dims of Q/K by angle m \theta_j with \theta_j = 10000^{-2j/d}, so the inner product of any positions m, n depends only on their relative distance m - n; ALiBi instead adds a linear bias -m \cdot |i - j| to... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/positional-encoding

Q: Why can't RoPE extrapolate directly to longer sequences (leading to PI/NTK/YaRN)?

【Positional Encoding RoPE/ALiBi 核心考点 5】Why can't RoPE extrapolate directly to longer sequences (leading to PI/NTK/YaRN)?。技术解析：Definition: RoPE (Rotary Position Embedding) rotates the 2j-th and 2j+1-th dims of Q/K by angle m \theta_j with \theta_j = 10000^{-2j/d}, so the inner product of any positions m, n depends only on their relative distance m - n; ALiBi instead adds a linear bias -m \cdot |i - j| to... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/positional-encoding

位置编码 RoPE/ALiBi

🎯Core Definition

RoPE (Rotary Position Embedding) rotates the

2j

-th and

2j+1

-th dims of Q/K by angle

m \theta_j

with

\theta_j = 10000^{-2j/d}

, so the inner product of any positions

m

n

depends only on their relative distance

m - n

; ALiBi instead adds a linear bias

-m \cdot |i - j|

to attention scores with zero parameters.

💡Use Cases

RoPE is used by essentially all mainstream models (LLaMA/Qwen/Mistral/Gemma; LLaMA-3 raises the base to 500000); ALiBi appears in BLOOM/MPT-style models that emphasize extrapolation and low cost; RoPE's formula, wavelength properties and extrapolation are staple interview topics.

⚡Key Problems Solved

learnable/absolute position embeddings fit a fixed length and fail beyond it; RoPE bakes position into a rotation that composes naturally with the dot product for relative positions, and its low-frequency dims rotate slowly (wavelength

\lambda_j = 2\pi \cdot 10000^{2j/d}

, lowest freq ~62832 tokens), giving a local "recency" prior; ALiBi needs no parameters and extrapolates natively, but its long-range relative discrimination is weak.

🎯5 High-Frequency Exam Points

RoPE rotation formula? Why does the inner product depend only on the relative offset

m - n

RoPE wavelength: period of the lowest-frequency dim? When does base=10000 cause periodic ambiguity (~62K tokens; LLaMA-3 uses 500000)?

RoPE vs absolute and learnable position embeddings?

How ALiBi works, and its pros and cons?

Why can't RoPE extrapolate directly to longer sequences (leading to PI/NTK/YaRN)?

📖 In-depth Guide:📄 transformer-architecture →

Updated 2026-08-12

🎯

Test Your Knowledge: Practice Questions for "Positional Encoding RoPE/ALiBi"

Single choice pitfall questions with instant feedback and mistake tracking.

🚀 Start Card Practice ➔

← Previous CardSSM & Hybrid Architectures Next Card →RoPE Extrapolation PI/NTK/YaRN

🔗 More LLMs Knowledge Cards

Agent & Tool Calling Alignment Tax & Preference Data Scaled Dot-Product Attention Attention Variants MHA/MQA/GQA