C1Easy★ Core Essential · Industrial BedrockPart C · Positional Encoding Evolution
Sinusoidal Positional Encoding
Classic Transformer fixed sine/cosine encodings across geometric wavelengths enabling linear relative shifts.
⏱️ Time:
O(S * D) 单次前向构造💾 Space:
O(S * D) 常驻只读常量表🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def sinusoidal_positional_encoding(seq_len: int, d_model: int) -> np.ndarray:
"""
纯手写 Sinusoidal 绝对位置编码。
参数:
seq_len: 序列最大长度 S
d_model: 隐藏特征维度 D (通常必须为偶数)
返回:
pe: (seq_len, d_model)
"""
assert d_model % 2 == 0, "d_model 必须为偶数"
# 1. 位置向量 pos: (seq_len, 1)
position =
# 2. 频率缩放向量 div_term: (d_model // 2,)
# 利用 exp(log) 保持高精数值稳定
i = np.arange(0, d_model, 2)
div_term = np.exp(-i * (np.log(10000.0) / d_model))
# 3. 构造输出矩阵
pe = np.zeros((seq_len, d_model), dtype=np.float32)
angles =
pe[:, 0::2] = np.sin(angles) # 偶数索引填 sin
pe[:, 1::2] = np.cos(angles) # 奇数索引填 cos
return pe
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why have modern LLMs moved away from additive sinusoidal embeddings toward multiplicative rotary embeddings (RoPE)?
💡
Core Mental Anchor / Mnemonic
Even dims take sin, odd dims take cos; log-exp frequencies map relative offsets to orthogonal rotations.
📐 Mathematical Derivation & Core Formula
### Geometric Wavelength Progression & Linear Relative Shift
Because self-attention is permutation-invariant, explicit position signals must be injected.
Vaswani et al. (2017) use alternating sine/cosine frequencies spanning wavelengths from to :
Key property:
For any fixed offset , is a linear orthogonal rotation of :
This allows attention heads to learn relative offsets via linear projections.
Because self-attention is permutation-invariant, explicit position signals must be injected.
Vaswani et al. (2017) use alternating sine/cosine frequencies spanning wavelengths from to :
Key property:
For any fixed offset , is a linear orthogonal rotation of :
This allows attention heads to learn relative offsets via linear projections.
🔄 Tensor Dimensions & Shape Flow
pos: (S, 1), div: (D/2,) -> angles: (S, D/2) -> interleave sin/cos -> pe: (S, D)
🛡️ Numerical Stability & Pitfalls Checklist
- Compute inverse frequencies as exp(-2i/d * ln 10000) to avoid power overflow/underflow issues.
- Original Transformer scales token embeddings by sqrt(d_model) before adding PE.
💻 Reference Code Implementation
import numpy as np
def sinusoidal_positional_encoding(seq_len: int, d_model: int) -> np.ndarray:
"""
纯手写 Sinusoidal 绝对位置编码。
参数:
seq_len: 序列最大长度 S
d_model: 隐藏特征维度 D (通常必须为偶数)
返回:
pe: (seq_len, d_model)
"""
assert d_model % 2 == 0, "d_model 必须为偶数"
# 1. 位置向量 pos: (seq_len, 1)
position = np.arange(seq_len)[:, np.newaxis]
# 2. 频率缩放向量 div_term: (d_model // 2,)
# 利用 exp(log) 保持高精数值稳定
i = np.arange(0, d_model, 2)
div_term = np.exp(-i * (np.log(10000.0) / d_model))
# 3. 构造输出矩阵
pe = np.zeros((seq_len, d_model), dtype=np.float32)
angles = position * div_term # 广播得到 (seq_len, d_model // 2)
pe[:, 0::2] = np.sin(angles) # 偶数索引填 sin
pe[:, 1::2] = np.cos(angles) # 奇数索引填 cos
return pe
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
pe = sinusoidal_positional_encoding(seq_len=10, d_model=16)
assert pe.shape == (10, 16)
# 检验第 0 个位置:sin(0)=0, cos(0)=1
assert np.isclose(pe[0, 0], 0.0) and np.isclose(pe[0, 1], 1.0)
print("✓ Sinusoidal 编码自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why have modern LLMs moved away from additive sinusoidal embeddings toward multiplicative rotary embeddings (RoPE)?
Additive absolute encodings are added only at the input embedding layer and can be diluted across many layers, whereas RoPE injects relative position directly into every attention layer's dot product and generalizes better across lengths.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.