E7EasyPart E · Loss Functions Handbook
Perplexity (PPL) Calculation
Exponentiated mean negative log-likelihood evaluating autoregressive language-model uncertainty.
⏱️ Time:
O(N)💾 Space:
O(1)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def calculate_perplexity(loss_values: np.ndarray) -> float:
"""
参数:
loss_values: 每个有效 Token 的负对数似然损失数组
返回:
PPL 标量
"""
mean_loss =
# 防指数上溢保护
if mean_loss > 50.0:
return float("inf")
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why can't you compare raw token-level Perplexity directly across models with different tokenizers?
💡
Core Mental Anchor / Mnemonic
Exponentiate valid-token average cross-entropy loss: PPL = exp(mean_CE).
📐 Mathematical Derivation & Core Formula
### Probabilistic Branching Factor Interpretation
Perplexity (PPL) measures how well an autoregressive language model predicts a held-out token sequence :
Intuitively, PPL represents the effective uniform branching factor (average number of equally likely choices) at each next-token step; lower PPL indicates stronger predictive certainty.
Perplexity (PPL) measures how well an autoregressive language model predicts a held-out token sequence :
Intuitively, PPL represents the effective uniform branching factor (average number of equally likely choices) at each next-token step; lower PPL indicates stronger predictive certainty.
🔄 Tensor Dimensions & Shape Flow
valid token NLLs (N,) -> np.mean -> mean_loss -> np.exp -> scalar PPL
🛡️ Numerical Stability & Pitfalls Checklist
- Exclude padded/
ignore_indexpositions before averaging cross-entropy losses. - Cap mean_loss (e.g., <= 100) before
expduring unstable runs to avoid float overflow.
💻 Reference Code Implementation
import numpy as np
def calculate_perplexity(loss_values: np.ndarray) -> float:
"""
参数:
loss_values: 每个有效 Token 的负对数似然损失数组
返回:
PPL 标量
"""
mean_loss = np.mean(loss_values)
# 防指数上溢保护
if mean_loss > 50.0:
return float("inf")
return float(np.exp(mean_loss))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
losses = np.array([0.0, 0.0]) # 完美确定
assert calculate_perplexity(losses) == 1.0
losses_rand = np.array([np.log(4.0)]) # 在 4 个词间随机猜测
assert np.isclose(calculate_perplexity(losses_rand), 4.0)
print("✓ Perplexity 困惑度自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why can't you compare raw token-level Perplexity directly across models with different tokenizers?
Tokenizers segment text at different granularities (e.g., 32k vs. 100k vocabularies pack different numbers of characters per token), making raw token-level PPL non-comparable. Cross-tokenizer evaluation normalizes to bits per byte (BPB) or character-level likelihood.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.