C4HardPart C · Positional Encoding Evolution

NTK-Aware Scaled RoPE Extrapolation

⚡Flashcard Drill➔

Rescale rotary base frequency non-uniformly via NTK intuition to preserve high-frequency local discrimination.

⏱️ Time: O(D) 初始化开销
💾 Space: O(D) 仅常数频率向量
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_ntk_scaled_frequencies(
dim: int,
alpha: float,
theta_base: float = 10000.0
) -> np.ndarray:
"""
NTK-Aware 动态基频缩放。
参数:
dim: 隐藏头维度 D
alpha: 长度扩展倍率 (例如从 4k 外推至 16k 时 alpha = 4.0)
theta_base: 原始基频底数 (通常 10000.0)
返回:
scaled_freqs: (dim // 2,)
"""
half_dim =
# 核心公式: base' = base * alpha ** (dim / (dim - 2))
base_prime = theta_base * (alpha ** (dim / (dim - 2)))
# 基于新的底数计算缩放后的各通道角频率
i = np.arange(0, half_dim)
scaled_freqs =
return scaled_freqs
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What additional problem does YaRN solve beyond NTK-Aware RoPE scaling?
Next: Scaled Dot-Product Attention with Causal Mask →
💡

Core Mental Anchor / Mnemonic

Stretch slow frequencies for long context while keeping fast frequencies sharp for nearby tokens.

📐 Mathematical Derivation & Core Formula

b′=b⋅αdd−2,θi′=1(b′)2i/db' = b \cdot \alpha^{\frac{d}{d - 2}}, \quad \theta_i' = \frac{1}{(b')^{2i/d}}
### Limits of Linear Interpolation & NTK Frequency Scaling
When extending a model trained at context length LL to L′=αLL' = \alpha L (α>1\alpha > 1):
- Direct extrapolation: Unseen phase angles cause attention collapse and rapid perplexity spikes;
- Linear Position Interpolation (PI): Scales pos←pos/αpos \leftarrow pos / \alpha, compressing high-frequency wavelengths and blurring adjacent local tokens;
- NTK-Aware scaling: Stretches low-frequency (global) dimensions aggressively while preserving high-frequency (local) resolution by adjusting the rotary base frequency b→b′b \to b' non-uniformly across dimensions.

🔄 Tensor Dimensions & Shape Flow

scale alpha -> compute adjusted base_prime -> recompute inverse frequencies scaled_freqs: (D//2,)

🛡️ Numerical Stability & Pitfalls Checklist

  • Bypass scaling when alpha <= 1 so short contexts retain native base frequencies.
  • YaRN builds on NTK scaling by adjusting Softmax temperature to stabilize entropy on very long contexts.
  • Dynamic NTK adapts alpha at inference time based on active sequence length.

💻 Reference Code Implementation

import numpy as np

def compute_ntk_scaled_frequencies(
    dim: int,
    alpha: float,
    theta_base: float = 10000.0
) -> np.ndarray:
    """
    NTK-Aware 动态基频缩放。
    参数:
        dim: 隐藏头维度 D
        alpha: 长度扩展倍率 (例如从 4k 外推至 16k 时 alpha = 4.0)
        theta_base: 原始基频底数 (通常 10000.0)
    返回:
        scaled_freqs: (dim // 2,)
    """
    half_dim = dim // 2
    # 核心公式: base' = base * alpha ** (dim / (dim - 2))
    base_prime = theta_base * (alpha ** (dim / (dim - 2)))
    # 基于新的底数计算缩放后的各通道角频率
    i = np.arange(0, half_dim)
    scaled_freqs = 1.0 / (base_prime ** (i / half_dim))
    return scaled_freqs

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
freq_orig = 1.0 / (10000.0 ** (np.arange(0, 32) / 32))
freq_ntk = compute_ntk_scaled_frequencies(dim=64, alpha=4.0, theta_base=10000.0)
assert len(freq_ntk) == 32
# 高频通道(i=0)缩放极其微弱,保持相邻区分力
assert np.isclose(freq_ntk[0], freq_orig[0], rtol=0.1)
# 低频通道(i=31)显著压缩,适应长程全局
assert freq_ntk[-1] < freq_orig[-1]
print("✓ NTK-Aware RoPE 频率缩放自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What additional problem does YaRN solve beyond NTK-Aware RoPE scaling?
As sequence length grows, more tokens compete inside Softmax, inflating attention entropy and over-smoothing distributions. YaRN introduces an attention temperature scale factor t\sqrt{t} applied to QK⊤/dQK^\top / \sqrt{d} to restore sharper attention focus.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Attention with Linear Biases (ALiBi)Back to Part C & All 105Next: Scaled Dot-Product Attention with Causal Mask →