C1Easy★ Core Essential · Industrial BedrockPart C · Positional Encoding Evolution

Sinusoidal Positional Encoding

⚡Flashcard Drill➔

Classic Transformer fixed sine/cosine encodings across geometric wavelengths enabling linear relative shifts.

⏱️ Time: O(S * D) 单次前向构造
💾 Space: O(S * D) 常驻只读常量表
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def sinusoidal_positional_encoding(seq_len: int, d_model: int) -> np.ndarray:
"""
纯手写 Sinusoidal 绝对位置编码。
参数:
seq_len: 序列最大长度 S
d_model: 隐藏特征维度 D (通常必须为偶数)
返回:
pe: (seq_len, d_model)
"""
assert d_model % 2 == 0, "d_model 必须为偶数"
# 1. 位置向量 pos: (seq_len, 1)
position =
# 2. 频率缩放向量 div_term: (d_model // 2,)
# 利用 exp(log) 保持高精数值稳定
i = np.arange(0, d_model, 2)
div_term = np.exp(-i * (np.log(10000.0) / d_model))
# 3. 构造输出矩阵
pe = np.zeros((seq_len, d_model), dtype=np.float32)
angles =
pe[:, 0::2] = np.sin(angles) # 偶数索引填 sin
pe[:, 1::2] = np.cos(angles) # 奇数索引填 cos
return pe
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why have modern LLMs moved away from additive sinusoidal embeddings toward multiplicative rotary embeddings (RoPE)?
Next: Rotary Position Embedding (RoPE) →
💡

Core Mental Anchor / Mnemonic

Even dims take sin, odd dims take cos; log-exp frequencies map relative offsets to orthogonal rotations.

📐 Mathematical Derivation & Core Formula

PE(pos,2i)=sin⁡(pos100002i/d),PE(pos,2i+1)=cos⁡(pos100002i/d)\mathrm{PE}_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad \mathrm{PE}_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)
### Geometric Wavelength Progression & Linear Relative Shift
Because self-attention is permutation-invariant, explicit position signals must be injected.
Vaswani et al. (2017) use alternating sine/cosine frequencies spanning wavelengths from 2π2\pi to 10000⋅2π10000 \cdot 2\pi:
ωi=1100002i/d=exp⁡(−2idln⁡10000)\omega_i = \frac{1}{10000^{2i/d}} = \exp\left(-\frac{2i}{d} \ln 10000\right)
Key property:
For any fixed offset kk, PEpos+k\mathrm{PE}_{pos+k} is a linear orthogonal rotation of PEpos\mathrm{PE}_{pos}:
[sin⁡(ωi(pos+k))cos⁡(ωi(pos+k))]=[cos⁡(ωik)sin⁡(ωik)−sin⁡(ωik)cos⁡(ωik)][sin⁡(ωipos)cos⁡(ωipos)]\begin{bmatrix} \sin(\omega_i(pos+k)) \\ \cos(\omega_i(pos+k)) \end{bmatrix} = \begin{bmatrix} \cos(\omega_i k) & \sin(\omega_i k) \\ -\sin(\omega_i k) & \cos(\omega_i k) \end{bmatrix} \begin{bmatrix} \sin(\omega_i pos) \\ \cos(\omega_i pos) \end{bmatrix}
This allows attention heads to learn relative offsets via linear projections.

🔄 Tensor Dimensions & Shape Flow

pos: (S, 1), div: (D/2,) -> angles: (S, D/2) -> interleave sin/cos -> pe: (S, D)

🛡️ Numerical Stability & Pitfalls Checklist

  • Compute inverse frequencies as exp(-2i/d * ln 10000) to avoid power overflow/underflow issues.
  • Original Transformer scales token embeddings by sqrt(d_model) before adding PE.

💻 Reference Code Implementation

import numpy as np

def sinusoidal_positional_encoding(seq_len: int, d_model: int) -> np.ndarray:
    """
    纯手写 Sinusoidal 绝对位置编码。
    参数:
        seq_len: 序列最大长度 S
        d_model: 隐藏特征维度 D (通常必须为偶数)
    返回:
        pe: (seq_len, d_model)
    """
    assert d_model % 2 == 0, "d_model 必须为偶数"
    
    # 1. 位置向量 pos: (seq_len, 1)
    position = np.arange(seq_len)[:, np.newaxis]
    
    # 2. 频率缩放向量 div_term: (d_model // 2,)
    # 利用 exp(log) 保持高精数值稳定
    i = np.arange(0, d_model, 2)
    div_term = np.exp(-i * (np.log(10000.0) / d_model))
    
    # 3. 构造输出矩阵
    pe = np.zeros((seq_len, d_model), dtype=np.float32)
    angles = position * div_term  # 广播得到 (seq_len, d_model // 2)
    
    pe[:, 0::2] = np.sin(angles)  # 偶数索引填 sin
    pe[:, 1::2] = np.cos(angles)  # 奇数索引填 cos
    return pe

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
pe = sinusoidal_positional_encoding(seq_len=10, d_model=16)
assert pe.shape == (10, 16)
# 检验第 0 个位置:sin(0)=0, cos(0)=1
assert np.isclose(pe[0, 0], 0.0) and np.isclose(pe[0, 1], 1.0)
print("✓ Sinusoidal 编码自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why have modern LLMs moved away from additive sinusoidal embeddings toward multiplicative rotary embeddings (RoPE)?
Additive absolute encodings are added only at the input embedding layer and can be diluted across many layers, whereas RoPE injects relative position directly into every attention layer's QK⊤Q K^\top dot product and generalizes better across lengths.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Inverted Dropout with Train/Eval ScalingBack to Part C & All 105Next: Rotary Position Embedding (RoPE) →