C4HardPart C · Positional Encoding Evolution
NTK-Aware Scaled RoPE Extrapolation
Rescale rotary base frequency non-uniformly via NTK intuition to preserve high-frequency local discrimination.
⏱️ Time:
O(D) 初始化开销💾 Space:
O(D) 仅常数频率向量🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_ntk_scaled_frequencies(
dim: int,
alpha: float,
theta_base: float = 10000.0
) -> np.ndarray:
"""
NTK-Aware 动态基频缩放。
参数:
dim: 隐藏头维度 D
alpha: 长度扩展倍率 (例如从 4k 外推至 16k 时 alpha = 4.0)
theta_base: 原始基频底数 (通常 10000.0)
返回:
scaled_freqs: (dim // 2,)
"""
half_dim =
# 核心公式: base' = base * alpha ** (dim / (dim - 2))
base_prime = theta_base * (alpha ** (dim / (dim - 2)))
# 基于新的底数计算缩放后的各通道角频率
i = np.arange(0, half_dim)
scaled_freqs =
return scaled_freqs
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What additional problem does YaRN solve beyond NTK-Aware RoPE scaling?
💡
Core Mental Anchor / Mnemonic
Stretch slow frequencies for long context while keeping fast frequencies sharp for nearby tokens.
📐 Mathematical Derivation & Core Formula
### Limits of Linear Interpolation & NTK Frequency Scaling
When extending a model trained at context length to ():
- Direct extrapolation: Unseen phase angles cause attention collapse and rapid perplexity spikes;
- Linear Position Interpolation (PI): Scales , compressing high-frequency wavelengths and blurring adjacent local tokens;
- NTK-Aware scaling: Stretches low-frequency (global) dimensions aggressively while preserving high-frequency (local) resolution by adjusting the rotary base frequency non-uniformly across dimensions.
When extending a model trained at context length to ():
- Direct extrapolation: Unseen phase angles cause attention collapse and rapid perplexity spikes;
- Linear Position Interpolation (PI): Scales , compressing high-frequency wavelengths and blurring adjacent local tokens;
- NTK-Aware scaling: Stretches low-frequency (global) dimensions aggressively while preserving high-frequency (local) resolution by adjusting the rotary base frequency non-uniformly across dimensions.
🔄 Tensor Dimensions & Shape Flow
scale alpha -> compute adjusted base_prime -> recompute inverse frequencies scaled_freqs: (D//2,)
🛡️ Numerical Stability & Pitfalls Checklist
- Bypass scaling when alpha <= 1 so short contexts retain native base frequencies.
- YaRN builds on NTK scaling by adjusting Softmax temperature to stabilize entropy on very long contexts.
- Dynamic NTK adapts alpha at inference time based on active sequence length.
💻 Reference Code Implementation
import numpy as np
def compute_ntk_scaled_frequencies(
dim: int,
alpha: float,
theta_base: float = 10000.0
) -> np.ndarray:
"""
NTK-Aware 动态基频缩放。
参数:
dim: 隐藏头维度 D
alpha: 长度扩展倍率 (例如从 4k 外推至 16k 时 alpha = 4.0)
theta_base: 原始基频底数 (通常 10000.0)
返回:
scaled_freqs: (dim // 2,)
"""
half_dim = dim // 2
# 核心公式: base' = base * alpha ** (dim / (dim - 2))
base_prime = theta_base * (alpha ** (dim / (dim - 2)))
# 基于新的底数计算缩放后的各通道角频率
i = np.arange(0, half_dim)
scaled_freqs = 1.0 / (base_prime ** (i / half_dim))
return scaled_freqs
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
freq_orig = 1.0 / (10000.0 ** (np.arange(0, 32) / 32))
freq_ntk = compute_ntk_scaled_frequencies(dim=64, alpha=4.0, theta_base=10000.0)
assert len(freq_ntk) == 32
# 高频通道(i=0)缩放极其微弱,保持相邻区分力
assert np.isclose(freq_ntk[0], freq_orig[0], rtol=0.1)
# 低频通道(i=31)显著压缩,适应长程全局
assert freq_ntk[-1] < freq_orig[-1]
print("✓ NTK-Aware RoPE 频率缩放自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What additional problem does YaRN solve beyond NTK-Aware RoPE scaling?
As sequence length grows, more tokens compete inside Softmax, inflating attention entropy and over-smoothing distributions. YaRN introduces an attention temperature scale factor applied to to restore sharper attention focus.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.