Long-Context Extension

Long-context extension scales the effective context from 4K/8K to 128K/1M, via two main routes. Route one: positional-encoding surgery on RoPE — PI (Positional Interpolation) scales position m by $1/s$ for linear interpolation; NTK-aware scaling down-weights only low frequencies while preserving high ones; YaRN (Yet another RoPE extensioN) adds a temperature t on top of NTK, scaling attention logits by $1/\sqrt{t}$ — the most widely used today. Route two: continued pretraining on long documents (books, code repos) packed in sequence at a small learning rate.

long-document QA, codebase understanding, long-horizon agent tasks; inference relies on KV-cache management (see the AI_Infra module).

naive extrapolation pushes RoPE positions beyond the training range, destabilizing attention scores and exploding perplexity. PI/NTK/YaRN extend 4K to 128K with no or minimal training (e.g., YaRN-LLaMA-2-128K) while perplexity stays on par with short contexts; the cost is KV-cache memory growing linearly with length: a 128K-context 70B model needs hundreds of GB of KV cache (about $2 \times 2 \times L \times H \times d \times s \times b$), requiring PagedAttention-style management.

Test Your Knowledge: Practice Questions for "Long-Context Extension"

Single choice pitfall questions with instant feedback and mistake tracking.