M4-027M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Medium
⚡Flashcard Drill➔Mastery:

Comparing Positional Encodings: Absolute vs Relative vs RoPE vs ALiBi

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-028 →
📐 Mathematical Definition
Abs: xi←xi+pi;Rel: biasij=f(i−j);ALiBi: biasij=−mh⋅(i−j)\text{Abs}:\ x_i\leftarrow x_i+p_i;\qquad \text{Rel}:\ \text{bias}_{ij}=f(i-j);\qquad \text{ALiBi}:\ \text{bias}_{ij}=-m_h\cdot(i-j)
⚡ One-Sentence Core Answer
Absolute PE adds coordinates to inputs (poor extrapolation); Relative PE learns offset biases; RoPE rotates queries and keys; ALiBi adds non-learned linear penalties with optimal extrapolation.

📌 Key Takeaways

  • •
    Absolute (BERT/GPT-2): adds position vectors x+Px + P; strictly bounded by maximum training sequence length
  • •
    Relative (T5): adds learned bias matrix Bi,j=b(i−j)B_{i, j} = b(i - j) directly into attention logits; memory intensive (O(N2)O(N^2))
  • •
    RoPE (LLaMA): multiplicative rotation of Q and K; elegant, hardware efficient, standard across foundation LLMs
  • •
    ALiBi (Press et al.): adds non-learned static linear slope penalty qiTkj−m⋅∣i−j∣q_i^T k_j - m \cdot |i - j|; strongest zero-shot length extrapolation

📐 Mathematical Derivations

Comparative Formulations:
① Absolute Learned PE (GPT-2):
h0=Embed(x)+Wpos[pos]h_0 = \text{Embed}(x) + W_{\text{pos}}[pos]. Strictly fails for pos>Ltrainpos > L_{\text{train}}.
② Relative Position Bias (T5 / Raffel et al.):
Attention(Q,K,V)=softmax(QKTd+Brel)V\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d}} + B_{\text{rel}} \right) V, where Bi,j=f(i−j)B_{i, j} = f(i - j) is a learned bucketed bias.
- Flaw: Materializing Brel∈RB×H×N×NB_{\text{rel}} \in \mathbb{R}^{B \times H \times N \times N} requires massive GPU VRAM bandwidth, slowing down inference.
③ Rotary Position Embedding (RoPE):
Attention(Q,K,V)=softmax((RmQ)(RnK)Td)V\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{(R_m Q) (R_n K)^T}{\sqrt{d}} \right) V. Incurs zero memory overhead, compatible with KV caching.
④ ALiBi (Attention with Linear Biases / Press et al., ICLR 2022):
Softmax(qiTkjd−m⋅∣i−j∣)\text{Softmax}\left( \frac{q_i^T k_j}{\sqrt{d}} - m \cdot |i - j| \right), where m=2−8hHm = 2^{-\frac{8 h}{H}} is a fixed geometric slope per head.
Zero learned parameters. Models trained on 2,048 tokens extrapolate zero-shot to 8,192 tokens with stable perplexity.

🏭 Production Trade-offs

Why Industry Chose RoPE: While ALiBi achieves exceptional zero-shot extrapolation, RoPE combined with modern context extension techniques (NTK-aware, YaRN) delivers superior in-distribution perplexity and higher semantic retrieval precision.
⚠️ Common Interview Pitfalls
  • ✕
    Using absolute learned positional embeddings for models intended to be fine-tuned on longer context windows
  • ✕
    Assuming ALiBi requires trainable parameters; ALiBi slopes mm are static, non-learned constants
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does ALiBi's linear penalty slope m=2−8h/Hm = 2^{-8h/H} vary across attention heads in a geometric progression?
  • ?
    What factors led open-source LLM developers to universally prefer RoPE over ALiBi despite ALiBi's extrapolation benefits?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-026: Rotary Position Embedding (RoPE): Mathematical Mechanics and Relative Distance Invariance📋 Back to BankNext →M4-028: Context Length Extension in RoPE: Position Interpolation, NTK-Aware Scaling, and YaRN