C3MediumPart C · Positional Encoding Evolution
Attention with Linear Biases (ALiBi)
Industrial-grade implementation and mathematical foundations of Attention with Linear Biases (ALiBi).
⏱️ Time Complexity:
O(S^2) 一次性构建偏置网格💾 Space Complexity:
O(H * S^2) 注意力矩阵偏置💡
Core Mental Anchor / Mnemonic
Master Attention with Linear Biases (ALiBi): enforce numerical stability, check tensor shapes, and eliminate redundant memory allocations.
📐 Mathematical Derivation & Core Formula
### Mathematical Derivation & Theoretical Principles
Detailed first-principles formulation and architectural mechanics for Attention with Linear Biases (ALiBi).
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
Detailed first-principles formulation and architectural mechanics for Attention with Linear Biases (ALiBi).
Refer to the LaTeX equation above for the core operator definition. The operator is designed to ensure strict numerical bounds, avoiding floating-point overflows and gradient anomalies.
🔄 Tensor Dimensions & Shape Flow
slopes: (H,) -> distance: (S, S) -> bias: (1, H, S, S) -> 直接加到 (B, H, S, S) 注意力分数矩阵
🛡️ Industrial Numerical Stability & Pitfalls
- Ensure proper multi-dimensional tensor broadcasting and keepdims retention.
- Enforce numerical guards (eps clamping and overflow thresholds) during exponentiation and division.
- Verify train versus eval mode behavioral distinctions (e.g. frozen running statistics and dropout bypass).
💻 Industrial Code Implementation
import numpy as np
def get_alibi_slopes(num_heads: int) -> np.ndarray:
"""生成每个头的几何衰减斜率 m"""
def get_slopes_power_of_2(n):
start = (2 ** (-2 ** -(np.log2(n) - 3)))
ratio = start
return [start * (ratio ** i) for i in range(n)]
if np.log2(num_heads).is_integer():
return np.array(get_slopes_power_of_2(num_heads))
else:
# 非 2 的幂次向下寻找最近的幂
closest_power = 2 ** int(np.log2(num_heads))
slopes = get_slopes_power_of_2(closest_power)
# 补齐多出的头
extra_slopes = get_slopes_power_of_2(2 * closest_power)[0::2][:num_heads - closest_power]
return np.array(slopes + extra_slopes)
def build_alibi_bias(seq_len: int, num_heads: int) -> np.ndarray:
"""
构建 ALiBi 偏置矩阵。
返回形状: (1, num_heads, seq_len, seq_len)
"""
slopes = get_alibi_slopes(num_heads) # (H,)
# 构造因果距离矩阵: i - j
pos_i = np.arange(seq_len)[:, np.newaxis]
pos_j = np.arange(seq_len)[np.newaxis, :]
distance = np.maximum(0, pos_i - pos_j) # 下三角距离矩阵 (S, S)
# 广播相乘: (H, 1, 1) * (1, S, S) -> (H, S, S)
bias = -slopes[:, np.newaxis, np.newaxis] * distance[np.newaxis, :, :]
return bias[np.newaxis, ...] # (1, H, S, S)
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter to verify correctness:
import numpy as np
bias = build_alibi_bias(seq_len=4, num_heads=4)
assert bias.shape == (1, 4, 4, 4)
# 对角线上 i=j，偏置应为 0
assert np.allclose(np.diagonal(bias[0, 0]), 0.0)
# 距离越远，负值越大 (惩罚越重)
assert bias[0, 0, 3, 0] < bias[0, 0, 3, 2]
print("✓ ALiBi 偏置自测通过")🎯 Core Architecture Follow-up Q&A
Q1:What are the key trade-offs and memory bottlenecks when deploying Attention with Linear Biases (ALiBi) in high-throughput inference?
Memory bandwidth (HBM to SRAM I/O) is the primary latency factor. Fusing element-wise operations and avoiding intermediate tensor materialization significantly outperforms naive implementations.
Q2:How does Attention with Linear Biases (ALiBi) handle extreme numerical boundaries or precision reduction (FP16/BF16/INT8)?
Under low precision, operations must be upcasted to FP32 during accumulation to prevent underflow/overflow, followed by proper scaling and clamping before converting back to the target format.