M4-025M4: Sequences & TransformersPositional Embeddings (Sinusoidal, RoPE, ALiBi)Easy
⚡Flashcard Drill➔Mastery:

Sinusoidal Positional Encoding: Mathematical Mechanics and Theoretical Length Extrapolation

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-026 →
📐 Mathematical Definition
PE(p,2i)=sin⁡(p/100002i/d),PE(p,2i+1)=cos⁡(p/100002i/d)PE_{(p,2i)}=\sin(p/10000^{2i/d}),\quad PE_{(p,2i+1)}=\cos(p/10000^{2i/d})
⚡ One-Sentence Core Answer
Sinusoidal encoding assigns geometric frequency waves across dimensions; because relative offsets can be expressed as linear transformations, it theoretically generalizes to unseen positions.

📌 Key Takeaways

  • •
    Formulation: PE(pos,2i)=sin⁡(pos/100002i/d)\text{PE}_{(pos, 2i)} = \sin(pos / 10000^{2i/d}), PE(pos,2i+1)=cos⁡(pos/100002i/d)\text{PE}_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})
  • •
    Linear shift property: there exists a linear rotation matrix MΔM_{\Delta} such that PEpos+Δ=MΔPEpos\text{PE}_{pos + \Delta} = M_{\Delta} \text{PE}_{pos}
  • •
    Extrapolation limitation: while mathematically defined for all pos∈[0,∞)pos \in [0, \infty), empirical attention fails on unseen out-of-distribution position coordinates

📐 Mathematical Derivations

Mathematical Formulations (Vaswani et al., 2017):
For position pospos and dimension index i∈[0,d/2−1]i \in [0, d/2 - 1], define frequency ωi=1100002i/d\omega_i = \frac{1}{10000^{2i/d}}.
PE(pos,2i)=sin⁡(pos⋅ωi)\text{PE}(pos, 2i) = \sin(pos \cdot \omega_i), PE(pos,2i+1)=cos⁡(pos⋅ωi)\quad \text{PE}(pos, 2i+1) = \cos(pos \cdot \omega_i).
Proof of Linear Relative Shift Property:
By standard trigonometric angle-addition identities:
sin⁡((pos+Δ)ωi)=sin⁡(pos⋅ωi)cos⁡(Δ⋅ωi)+cos⁡(pos⋅ωi)sin⁡(Δ⋅ωi)\sin((pos + \Delta) \omega_i) = \sin(pos \cdot \omega_i) \cos(\Delta \cdot \omega_i) + \cos(pos \cdot \omega_i) \sin(\Delta \cdot \omega_i),
cos⁡((pos+Δ)ωi)=cos⁡(pos⋅ωi)cos⁡(Δ⋅ωi)−sin⁡(pos⋅ωi)sin⁡(Δ⋅ωi)\cos((pos + \Delta) \omega_i) = \cos(pos \cdot \omega_i) \cos(\Delta \cdot \omega_i) - \sin(pos \cdot \omega_i) \sin(\Delta \cdot \omega_i).
In matrix form for each 2D frequency subspace:
[PE(pos+Δ,2i)PE(pos+Δ,2i+1)]=[cos⁡(Δωi)sin⁡(Δωi)−sin⁡(Δωi)cos⁡(Δωi)][PE(pos,2i)PE(pos,2i+1)]=R(Δωi)[PE(pos,2i)PE(pos,2i+1)]\begin{bmatrix} \text{PE}(pos + \Delta, 2i) \\ \text{PE}(pos + \Delta, 2i+1) \end{bmatrix} = \begin{bmatrix} \cos(\Delta \omega_i) & \sin(\Delta \omega_i) \\ -\sin(\Delta \omega_i) & \cos(\Delta \omega_i) \end{bmatrix} \begin{bmatrix} \text{PE}(pos, 2i) \\ \text{PE}(pos, 2i+1) \end{bmatrix} = R(\Delta \omega_i) \begin{bmatrix} \text{PE}(pos, 2i) \\ \text{PE}(pos, 2i+1) \end{bmatrix}.
Because R(Δωi)R(\Delta \omega_i) is independent of absolute position pospos, the self-attention mechanism can theoretically learn to attend based on relative distance Δ\Delta via linear projections.

🏭 Production Trade-offs

Theory vs Practice: Although mathematically defined for arbitrary sequence lengths, empirical extrapolation fails because high-dimensional attention weights overfit to the specific coordinate range seen during training. Modern LLMs have replaced sinusoidal encodings with RoPE.
⚠️ Common Interview Pitfalls
  • ✕
    Assuming sinusoidal position embeddings extrapolate seamlessly to 2×2\times training length in practice without fine-tuning
  • ✕
    Adding positional encodings after the first LayerNorm instead of directly to initial token embeddings
🎯 Interviewer Follow-up Inquiries
  • ?
    Why does the inner product of two sinusoidal position encodings decay as the distance between them increases?
  • ?
    How did RoPE directly adapt the 2D rotation matrix from sinusoidal encodings into multiplicative query-key rotations?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-024: Transformer Training Stability Techniques: QK-Norm, Logit Soft-Capping, and Auxiliary Z-Loss📋 Back to BankNext →M4-026: Rotary Position Embedding (RoPE): Mathematical Mechanics and Relative Distance Invariance