M4-032M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Medium
⚡Flashcard Drill➔Mastery:

ALiBi (Attention with Linear Biases): Mechanisms and Zero-Shot Length Extrapolation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-033 →
📐 Mathematical Definition
logitsij+=−mh⋅(i−j),mh=2−8h/H (geometric)\mathrm{logits}_{ij}\mathrel{+}=-m_h\cdot(i-j),\qquad m_h=2^{-8h/H}\ (\text{geometric})
⚡ One-Sentence Core Answer
ALiBi injects non-learned linear distance penalties directly into attention logits, allowing models trained on 2K tokens to extrapolate zero-shot to 8K+ tokens with stable perplexity.

📌 Key Takeaways

  • •
    Formulation: softmax(qiTkjd−m⋅∣i−j∣)V\text{softmax}\left( \frac{q_i^T k_j}{\sqrt{d}} - m \cdot |i - j| \right) V; subtracts distance penalty proportional to token separation
  • •
    Geometric head slopes: head slope m=2−8h/Hm = 2^{-8h/H}; assigns steep penalties to some heads (local context) and shallow penalties to others (broad context)
  • •
    Zero extra parameters: completely eliminates positional embedding matrices and out-of-distribution coordinate failures

📐 Mathematical Derivations

Mathematical Formulations (Press, Smith, Lewis, ICLR 2022):
In standard self-attention, positional information is either added to embeddings (x+Px + P) or multiplied into queries and keys (RmqR_m q). Both approaches evaluate neural projections on unseen coordinate domains during long-sequence inference, causing immediate loss explosion.
ALiBi Formulation:
Drop all positional embeddings from input tokens. Modify attention logits directly:
Ai,j=softmax(qikjTdk−m⋅∣i−j∣)A_{i, j} = \text{softmax}\left( \frac{q_i k_j^T}{\sqrt{d_k}} - m \cdot |i - j| \right), where i,ji, j are token positions and m>0m > 0 is a head-specific scalar slope.
- Geometric Progression of Slopes:
For a model with HH attention heads, slopes mm are fixed powers of 2:
m∈{2−8/H,2−16/H,…,2−8}m \in \left\{ 2^{-8/H}, 2^{-16/H}, \dots, 2^{-8} \right\}.
- Heads with large $m$: Penalize distant tokens severely, forcing the head to act as a localized nn-gram feature extractor.
- Heads with tiny $m$: Place negligible penalties on distance, attending broadly across the entire sequence.
- Extrapolation Mechanism: When sequence length expands from 2,048 to 16,384, the linear penalty −m∣i−j∣-m |i - j| simply scales naturally. No weights encounter out-of-distribution values, enabling zero-shot context expansion.

🏭 Production Trade-offs

Why RoPE is preferred over ALiBi in frontier LLMs: While ALiBi exhibits exceptional zero-shot extrapolation, RoPE provides superior long-distance information retrieval (e.g., Needle In A Haystack tasks) because ALiBi's monotonic penalty unconditionally suppresses distant retrieval.
⚠️ Common Interview Pitfalls
  • ✕
    Attempting to make ALiBi slopes mm learnable; making mm learnable destroys zero-shot extrapolation guarantees
  • ✕
    Using ALiBi for tasks requiring exact positional recall at distant offsets, where linear penalties suppress true signals
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does ALiBi struggle with Needle-In-A-Haystack retrieval at the very beginning of a 100K document?
  • ?
    How does the geometric ratio 2−8/H2^{-8/H} distribute receptive fields across Transformer attention heads?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-031: RoPE Implementation Details: Complex Formulation, Fast Kernel Tricks, and Base Frequencies📋 Back to BankNext →M4-033: 2D and 3D Rotary Position Embeddings (RoPE) for Vision and Video Multimodal Models