RoPE Extrapolation PI/NTK/YaRN

three ways to extend RoPE from the training length to longer contexts at low cost: PI (Position Interpolation) linearly rescales position $m$ to $m / s$ ($s$ = scaling factor); NTK-aware changes the rotary base to $\theta = 10000 \cdot s^{\frac{d}{d-2}}$ so high-frequency dims stay mostly uninterpolated while low-frequency dims stretch; YaRN (NTK-by-parts + temperature) interpolates only dims whose wavelength is much longer than the context window, keeps high-frequency dims untouched, and divides attention logits by a temperature $t \approx \sqrt{s}$.

cheap post-training context extension, e.g. LLaMA-2 from 4K to 32K/64K/128K; YaRN is currently the most used and best-performing option (adopted by many long-context open models), and comparing the three is a classic interview question.

naive extrapolation misreads the rotation periods of low-frequency dims beyond the trained length, collapsing perplexity; plain PI compresses every dimension uniformly, destroying high-frequency resolution and short-range dependencies; NTK-by-parts interpolates only dims with wavelength larger than the window, preserving local resolution, and YaRN adds a temperature to fix attention sharpness — at 16x extension it stays close to original perplexity, making it the best in quality and stability.

Why does direct RoPE extrapolation fail (periodic ambiguity, frequencies beyond training coverage)?

How PI works and its main flaw (uniform scaling loses high-frequency resolution)?

Difference between NTK-aware and NTK-by-parts? The wavelength criterion?

What are YaRN's two components? Role of temperature $t \approx \sqrt{s}$?

How to evaluate extrapolation methods (long-doc perplexity + needle precision recall)?