M4-027M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Medium
⚡Flashcard Drill➔Mastery:
Comparing Positional Encodings: Absolute vs Relative vs RoPE vs ALiBi
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Absolute PE adds coordinates to inputs (poor extrapolation); Relative PE learns offset biases; RoPE rotates queries and keys; ALiBi adds non-learned linear penalties with optimal extrapolation.
📌 Key Takeaways
- •Absolute (BERT/GPT-2): adds position vectors ; strictly bounded by maximum training sequence length
- •Relative (T5): adds learned bias matrix directly into attention logits; memory intensive ()
- •RoPE (LLaMA): multiplicative rotation of Q and K; elegant, hardware efficient, standard across foundation LLMs
- •ALiBi (Press et al.): adds non-learned static linear slope penalty ; strongest zero-shot length extrapolation
📐 Mathematical Derivations
Comparative Formulations:
① Absolute Learned PE (GPT-2):
. Strictly fails for .
② Relative Position Bias (T5 / Raffel et al.):
, where is a learned bucketed bias.
- Flaw: Materializing requires massive GPU VRAM bandwidth, slowing down inference.
③ Rotary Position Embedding (RoPE):
. Incurs zero memory overhead, compatible with KV caching.
④ ALiBi (Attention with Linear Biases / Press et al., ICLR 2022):
, where is a fixed geometric slope per head.
Zero learned parameters. Models trained on 2,048 tokens extrapolate zero-shot to 8,192 tokens with stable perplexity.
① Absolute Learned PE (GPT-2):
. Strictly fails for .
② Relative Position Bias (T5 / Raffel et al.):
, where is a learned bucketed bias.
- Flaw: Materializing requires massive GPU VRAM bandwidth, slowing down inference.
③ Rotary Position Embedding (RoPE):
. Incurs zero memory overhead, compatible with KV caching.
④ ALiBi (Attention with Linear Biases / Press et al., ICLR 2022):
, where is a fixed geometric slope per head.
Zero learned parameters. Models trained on 2,048 tokens extrapolate zero-shot to 8,192 tokens with stable perplexity.
🏭 Production Trade-offs
Why Industry Chose RoPE: While ALiBi achieves exceptional zero-shot extrapolation, RoPE combined with modern context extension techniques (NTK-aware, YaRN) delivers superior in-distribution perplexity and higher semantic retrieval precision.
⚠️ Common Interview Pitfalls
- ✕Using absolute learned positional embeddings for models intended to be fine-tuned on longer context windows
- ✕Assuming ALiBi requires trainable parameters; ALiBi slopes are static, non-learned constants
🎯 Interviewer Follow-up Inquiries
- ?Why does ALiBi's linear penalty slope vary across attention heads in a geometric progression?
- ?What factors led open-source LLM developers to universally prefer RoPE over ALiBi despite ALiBi's extrapolation benefits?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚