M4-032M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Medium
⚡Flashcard Drill➔Mastery:
ALiBi (Attention with Linear Biases): Mechanisms and Zero-Shot Length Extrapolation
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
ALiBi injects non-learned linear distance penalties directly into attention logits, allowing models trained on 2K tokens to extrapolate zero-shot to 8K+ tokens with stable perplexity.
📌 Key Takeaways
- •Formulation: ; subtracts distance penalty proportional to token separation
- •Geometric head slopes: head slope ; assigns steep penalties to some heads (local context) and shallow penalties to others (broad context)
- •Zero extra parameters: completely eliminates positional embedding matrices and out-of-distribution coordinate failures
📐 Mathematical Derivations
Mathematical Formulations (Press, Smith, Lewis, ICLR 2022):
In standard self-attention, positional information is either added to embeddings () or multiplied into queries and keys (). Both approaches evaluate neural projections on unseen coordinate domains during long-sequence inference, causing immediate loss explosion.
ALiBi Formulation:
Drop all positional embeddings from input tokens. Modify attention logits directly:
, where are token positions and is a head-specific scalar slope.
- Geometric Progression of Slopes:
For a model with attention heads, slopes are fixed powers of 2:
.
- Heads with large $m$: Penalize distant tokens severely, forcing the head to act as a localized -gram feature extractor.
- Heads with tiny $m$: Place negligible penalties on distance, attending broadly across the entire sequence.
- Extrapolation Mechanism: When sequence length expands from 2,048 to 16,384, the linear penalty simply scales naturally. No weights encounter out-of-distribution values, enabling zero-shot context expansion.
In standard self-attention, positional information is either added to embeddings () or multiplied into queries and keys (). Both approaches evaluate neural projections on unseen coordinate domains during long-sequence inference, causing immediate loss explosion.
ALiBi Formulation:
Drop all positional embeddings from input tokens. Modify attention logits directly:
, where are token positions and is a head-specific scalar slope.
- Geometric Progression of Slopes:
For a model with attention heads, slopes are fixed powers of 2:
.
- Heads with large $m$: Penalize distant tokens severely, forcing the head to act as a localized -gram feature extractor.
- Heads with tiny $m$: Place negligible penalties on distance, attending broadly across the entire sequence.
- Extrapolation Mechanism: When sequence length expands from 2,048 to 16,384, the linear penalty simply scales naturally. No weights encounter out-of-distribution values, enabling zero-shot context expansion.
🏭 Production Trade-offs
Why RoPE is preferred over ALiBi in frontier LLMs: While ALiBi exhibits exceptional zero-shot extrapolation, RoPE provides superior long-distance information retrieval (e.g., Needle In A Haystack tasks) because ALiBi's monotonic penalty unconditionally suppresses distant retrieval.
⚠️ Common Interview Pitfalls
- ✕Attempting to make ALiBi slopes learnable; making learnable destroys zero-shot extrapolation guarantees
- ✕Using ALiBi for tasks requiring exact positional recall at distant offsets, where linear penalties suppress true signals
🎯 Interviewer Follow-up Inquiries
- ?Why does ALiBi struggle with Needle-In-A-Haystack retrieval at the very beginning of a 100K document?
- ?How does the geometric ratio distribute receptive fields across Transformer attention heads?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚