RoPE Extrapolation PI/NTK/YaRN

three ways to extend RoPE from the training length to longer contexts at low cost: PI (Position Interpolation) linearly rescales position $m$ to $m / s$ ($s$ = scaling factor); NTK-aware changes the rotary base to $\theta = 10000 \cdot s^{\frac{d}{d-2}}$ so high-frequency dims stay mostly uninterpolated while low-frequency dims stretch; YaRN (NTK-by-parts + temperature) interpolates only dims whose wavelength is much longer than the context window, keeps high-frequency dims untouched, and divides attention logits by a temperature $t \approx \sqrt{s}$.

cheap post-training context extension, e.g. LLaMA-2 from 4K to 32K/64K/128K; YaRN is currently the most used and best-performing option (adopted by many long-context open models), and comparing the three is a classic interview question.

naive extrapolation misreads the rotation periods of low-frequency dims beyond the trained length, collapsing perplexity; plain PI compresses every dimension uniformly, destroying high-frequency resolution and short-range dependencies; NTK-by-parts interpolates only dims with wavelength larger than the window, preserving local resolution, and YaRN adds a temperature to fix attention sharpness — at 16x extension it stays close to original perplexity, making it the best in quality and stability.

Test Your Knowledge: Practice Questions for "RoPE Extrapolation PI/NTK/YaRN"

Single choice pitfall questions with instant feedback and mistake tracking.