C3MediumPart C · Positional Encoding Evolution
Attention with Linear Biases (ALiBi)
Add fixed head-specific negative distance biases directly to QK^T logits without vector position embeddings.
⏱️ Time:
O(S^2) 一次性构建偏置网格💾 Space:
O(H * S^2) 注意力矩阵偏置🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def get_alibi_slopes(num_heads: int) -> np.ndarray:
"""生成每个头的几何衰减斜率 m"""
def get_slopes_power_of_2(n):
start =
ratio = start
return [start * (ratio ** i) for i in range(n)]
if np.log2(num_heads).is_integer():
return np.array(get_slopes_power_of_2(num_heads))
else:
# 非 2 的幂次向下寻找最近的幂
closest_power = 2 ** int(np.log2(num_heads))
slopes = get_slopes_power_of_2(closest_power)
# 补齐多出的头
extra_slopes = get_slopes_power_of_2(2 * closest_power)[0::2][:num_heads - closest_power]
return np.array(slopes + extra_slopes)
def build_alibi_bias(seq_len: int, num_heads: int) -> np.ndarray:
"""
构建 ALiBi 偏置矩阵。
返回形状: (1, num_heads, seq_len, seq_len)
"""
slopes = get_alibi_slopes(num_heads) # (H,)
# 构造因果距离矩阵: i - j
pos_i = np.arange(seq_len)[:, np.newaxis]
pos_j = np.arange(seq_len)[np.newaxis, :]
distance = np.maximum(0, pos_i - pos_j) # 下三角距离矩阵 (S, S)
# 广播相乘: (H, 1, 1) * (1, S, S) -> (H, S, S)
bias = -slopes[:, np.newaxis, np.newaxis] * distance[np.newaxis, :, :]
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What are the main advantages and limitations of ALiBi compared to RoPE?
💡
Core Mental Anchor / Mnemonic
Subtract geometric head slopes * distance directly from attention logits for parameter-free extrapolation.
📐 Mathematical Derivation & Core Formula
### Linear Distance Penalty Across Attention Heads
ALiBi (Press et al., 2021) skips vector position encodings and instead subtracts a linear distance bias directly from causal attention logits ().
To give different heads distinct local vs. global receptive fields, head slopes decay geometrically across heads:
Validated in models like BLOOM and MPT, enabling context-length extrapolation beyond training length at inference.
ALiBi (Press et al., 2021) skips vector position encodings and instead subtracts a linear distance bias directly from causal attention logits ().
To give different heads distinct local vs. global receptive fields, head slopes decay geometrically across heads:
Validated in models like BLOOM and MPT, enabling context-length extrapolation beyond training length at inference.
🔄 Tensor Dimensions & Shape Flow
slopes: (H,) * distance: (S, S) -> bias: (1, H, S, S) -> add to attention logits (B, H, S, S)
🛡️ Numerical Stability & Pitfalls Checklist
- Ensure bias penalties are negative and grow with distance |i - j| via positive slopes m.
- FlashAttention-2 kernels accept alibi_slopes directly in fused SRAM evaluation.
💻 Reference Code Implementation
import numpy as np
def get_alibi_slopes(num_heads: int) -> np.ndarray:
"""生成每个头的几何衰减斜率 m"""
def get_slopes_power_of_2(n):
start = (2 ** (-2 ** -(np.log2(n) - 3)))
ratio = start
return [start * (ratio ** i) for i in range(n)]
if np.log2(num_heads).is_integer():
return np.array(get_slopes_power_of_2(num_heads))
else:
# 非 2 的幂次向下寻找最近的幂
closest_power = 2 ** int(np.log2(num_heads))
slopes = get_slopes_power_of_2(closest_power)
# 补齐多出的头
extra_slopes = get_slopes_power_of_2(2 * closest_power)[0::2][:num_heads - closest_power]
return np.array(slopes + extra_slopes)
def build_alibi_bias(seq_len: int, num_heads: int) -> np.ndarray:
"""
构建 ALiBi 偏置矩阵。
返回形状: (1, num_heads, seq_len, seq_len)
"""
slopes = get_alibi_slopes(num_heads) # (H,)
# 构造因果距离矩阵: i - j
pos_i = np.arange(seq_len)[:, np.newaxis]
pos_j = np.arange(seq_len)[np.newaxis, :]
distance = np.maximum(0, pos_i - pos_j) # 下三角距离矩阵 (S, S)
# 广播相乘: (H, 1, 1) * (1, S, S) -> (H, S, S)
bias = -slopes[:, np.newaxis, np.newaxis] * distance[np.newaxis, :, :]
return bias[np.newaxis, ...] # (1, H, S, S)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
bias = build_alibi_bias(seq_len=4, num_heads=4)
assert bias.shape == (1, 4, 4, 4)
# 对角线上 i=j,偏置应为 0
assert np.allclose(np.diagonal(bias[0, 0]), 0.0)
# 距离越远,负值越大 (惩罚越重)
assert bias[0, 0, 3, 0] < bias[0, 0, 3, 2]
print("✓ ALiBi 偏置自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What are the main advantages and limitations of ALiBi compared to RoPE?
ALiBi adds zero learned parameters and extrapolates cleanly beyond training context without position interpolation, though its monotonic recency bias can constrain capturing very specific non-local dependencies compared to RoPE.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.