← Back to LLMs Mind Map
⚡ LLMsID: rope-extrapolation

RoPE Extrapolation PI/NTK/YaRN

RoPE 外推方案 PI/NTK/YaRN
🎯Core Definition
three ways to extend RoPE from the training length to longer contexts at low cost: PI (Position Interpolation) linearly rescales position mm to m/sm / s (ss = scaling factor); NTK-aware changes the rotary base to θ=10000⋅sdd−2\theta = 10000 \cdot s^{\frac{d}{d-2}} so high-frequency dims stay mostly uninterpolated while low-frequency dims stretch; YaRN (NTK-by-parts + temperature) interpolates only dims whose wavelength is much longer than the context window, keeps high-frequency dims untouched, and divides attention logits by a temperature t≈st \approx \sqrt{s}.
💡Use Cases
cheap post-training context extension, e.g. LLaMA-2 from 4K to 32K/64K/128K; YaRN is currently the most used and best-performing option (adopted by many long-context open models), and comparing the three is a classic interview question.
⚡Key Problems Solved
naive extrapolation misreads the rotation periods of low-frequency dims beyond the trained length, collapsing perplexity; plain PI compresses every dimension uniformly, destroying high-frequency resolution and short-range dependencies; NTK-by-parts interpolates only dims with wavelength larger than the window, preserving local resolution, and YaRN adds a temperature to fix attention sharpness — at 16x extension it stays close to original perplexity, making it the best in quality and stability.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of RoPE Extrapolation PI/NTK/YaRN.
Use Cases
In production engineering, what are the primary use cases for RoPE Extrapolation PI/NTK/YaRN and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Why does direct RoPE extrapolation fail (periodic ambiguity, frequencies beyond training coverage)?
2
How PI works and its main flaw (uniform scaling loses high-frequency resolution)?
3
Difference between NTK-aware and NTK-by-parts? The wavelength criterion?
4
What are YaRN's two components? Role of temperature t≈st \approx \sqrt{s}?
5
How to evaluate extrapolation methods (long-doc perplexity + needle precision recall)?
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "RoPE Extrapolation PI/NTK/YaRN"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardPositional Encoding RoPE/ALiBiNext Card →FFN & SwiGLU

🔗 More LLMs Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/rope-extrapolation
中文速览与面试要点

RoPE 外推方案 PI/NTK/YaRN (RoPE Extrapolation PI/NTK/YaRN)

定义: 三类把 RoPE 从训练长度低成本扩展到更长上下文的方案:PI (Position Interpolation) 把位置 m 线性缩放到 m / s(s 为扩展倍率);NTK-aware 把旋转基数改为 = 10000 s^ dd-2,让高频维度基本不插值、低频维度拉伸;YaRN(NTK-by-parts + 温度)按"波长与上下文窗口的关系"决定哪些维度插值(波长 窗口的维度插值,高频保留),并对注意力 logits 除以温度 t(约为 s)。 使用场景: 预训练后低成本扩展上下文,典型如把 LLaMA-2 的 4K 扩到 32K/64K/128K;YaRN 是目前最常用、效果最好的方案(长上下文开源模型普遍采用),面试必考三者区别。 解决的问题: 直接外推时位置超过训练覆盖,低频维度的旋转周期被"误解",困惑度崩坏;朴素 PI 把所有维度均匀压缩,高频分辨率被破坏、短距离依赖退化;NTK-by-parts 只插值波长大于窗口的维度,保留局部分辨率,YaRN 再配温度修正注意力锐度,16 倍扩展下仍保持接近原始困惑度,是三种方案中质量与稳定性最好的。

核心要点

  • 为什么 RoPE 直接外推会失效?(周期混淆、频率超出训练覆盖)
  • PI 的原理与主要缺陷?(均匀缩放损失高频分辨率)
  • NTK-aware 与 NTK-by-parts 的区别?波长判据是什么?
  • YaRN 由哪两部分组成?温度 t s 的作用?
  • 如何评估外推方案的效果?(长文档困惑度 + Needle 精确召回)