I2Medium★ Core Essential · Industrial BedrockPart I · Inference, Decoding & Quantization
LLM Generation Sampler (Temperature, Top-K & Top-P)
Three-stage LLM decoding sampler combining Temperature scaling, Top-K masking, and Top-P nucleus filtering.
⏱️ Time:
O(V log V) 排序开销,V 为词表大小💾 Space:
O(V)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def sample_next_token(
logits: np.ndarray, # (vocab_size,) 未归一化预测
temperature: float = 0.7,
top_k: int = 50,
top_p: float = 0.9
) -> int:
# 1. 贪心特例
if temperature == 0.0:
return
# 2. 温度缩放
scaled_logits = logits / temperature
# 3. Top-K 截断
if top_k > 0 and top_k < len(scaled_logits):
# 寻找第 top_k 个最大值的阈值
kth_val = np.partition(scaled_logits, -top_k)[-top_k]
scaled_logits[scaled_logits < kth_val] = -np.inf
# 4. Softmax 转换为有效概率分布
max_l = np.max(scaled_logits)
exp_l = np.exp(scaled_logits - max_l)
probs = exp_l / np.sum(exp_l)
# 5. Top-P (核采样) 截断
if top_p < 1.0:
sorted_indices = np.argsort(-probs)
sorted_probs = probs[sorted_indices]
cumulative_probs = np.cumsum(sorted_probs)
# 找到累计概率刚刚超过 top_p 的位置
cutoff_mask = cumulative_probs > top_p
# 保证至少保留第 1 个最高概率词
cutoff_mask[0] = False
# 剔除截断之外的词
filtered_indices = sorted_indices[cutoff_mask]
probs[filtered_indices] = 0.0
# 重新归一化
probs =
# 6. 多项分布随机采样抽签
return int(np.random.choice(len(probs), p=probs))
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is Min-P sampling, and what advantage does it have over Top-P?
💡
Core Mental Anchor / Mnemonic
Scale by temperature, keep top-K, truncate cumulative top-P mass (keeping >= 1 token), sample.
📐 Mathematical Derivation & Core Formula
### Three-Stage Logit Modulation & Truncation
1. Temperature ($T$): Scales logits as . As , Softmax sharpens into a Dirac delta / greedy argmax; flattens probabilities;
2. Top-$K$: Retains only the top logits and masks the rest to ;
3. Top-$P$ (Nucleus Sampling): Sorts probabilities descending and keeps the smallest prefix whose cumulative mass reaches (e.g., ), adapting candidate count to context entropy.
1. Temperature ($T$): Scales logits as . As , Softmax sharpens into a Dirac delta / greedy argmax; flattens probabilities;
2. Top-$K$: Retains only the top logits and masks the rest to ;
3. Top-$P$ (Nucleus Sampling): Sorts probabilities descending and keeps the smallest prefix whose cumulative mass reaches (e.g., ), adapting candidate count to context entropy.
🔄 Tensor Dimensions & Shape Flow
logits -> divide by T -> Top-K mask -inf -> Softmax -> cumulative Top-P mask & renorm -> sample token_id
🛡️ Numerical Stability & Pitfalls Checklist
- Handle
temperature == 0directly viaargmaxrather than dividing by zero. - Always retain at least the highest-probability token in Top-P so remaining probabilities never sum to zero.
💻 Reference Code Implementation
import numpy as np
def sample_next_token(
logits: np.ndarray, # (vocab_size,) 未归一化预测
temperature: float = 0.7,
top_k: int = 50,
top_p: float = 0.9
) -> int:
# 1. 贪心特例
if temperature == 0.0:
return int(np.argmax(logits))
# 2. 温度缩放
scaled_logits = logits / temperature
# 3. Top-K 截断
if top_k > 0 and top_k < len(scaled_logits):
# 寻找第 top_k 个最大值的阈值
kth_val = np.partition(scaled_logits, -top_k)[-top_k]
scaled_logits[scaled_logits < kth_val] = -np.inf
# 4. Softmax 转换为有效概率分布
max_l = np.max(scaled_logits)
exp_l = np.exp(scaled_logits - max_l)
probs = exp_l / np.sum(exp_l)
# 5. Top-P (核采样) 截断
if top_p < 1.0:
sorted_indices = np.argsort(-probs)
sorted_probs = probs[sorted_indices]
cumulative_probs = np.cumsum(sorted_probs)
# 找到累计概率刚刚超过 top_p 的位置
cutoff_mask = cumulative_probs > top_p
# 保证至少保留第 1 个最高概率词
cutoff_mask[0] = False
# 剔除截断之外的词
filtered_indices = sorted_indices[cutoff_mask]
probs[filtered_indices] = 0.0
# 重新归一化
probs = probs / np.sum(probs)
# 6. 多项分布随机采样抽签
return int(np.random.choice(len(probs), p=probs))
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
logits = np.array([10.0, 9.0, 1.0, 0.0])
# 贪心模式必定选第 0 个
assert sample_next_token(logits, temperature=0.0) == 0
# Top-K=1 等价于贪心
assert sample_next_token(logits, temperature=1.0, top_k=1) == 0
print("✓ 大模型生成采样器自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:What is Min-P sampling, and what advantage does it have over Top-P?
In low-entropy steps where one token clearly dominates, Top-P can still admit low-probability noise near the threshold tail. Min-P sets a relative floor (e.g., ), cleanly pruning tail noise when confident while preserving diversity when uncertain.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.