I2Medium★ Core Essential · Industrial BedrockPart I · Inference, Decoding & Quantization

LLM Generation Sampler (Temperature, Top-K & Top-P)

⚡Flashcard Drill➔

Three-stage LLM decoding sampler combining Temperature scaling, Top-K masking, and Top-P nucleus filtering.

⏱️ Time: O(V log V) 排序开销,V 为词表大小
💾 Space: O(V)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def sample_next_token(
logits: np.ndarray, # (vocab_size,) 未归一化预测
temperature: float = 0.7,
top_k: int = 50,
top_p: float = 0.9
) -> int:
# 1. 贪心特例
if temperature == 0.0:
return
# 2. 温度缩放
scaled_logits = logits / temperature
# 3. Top-K 截断
if top_k > 0 and top_k < len(scaled_logits):
# 寻找第 top_k 个最大值的阈值
kth_val = np.partition(scaled_logits, -top_k)[-top_k]
scaled_logits[scaled_logits < kth_val] = -np.inf
# 4. Softmax 转换为有效概率分布
max_l = np.max(scaled_logits)
exp_l = np.exp(scaled_logits - max_l)
probs = exp_l / np.sum(exp_l)
# 5. Top-P (核采样) 截断
if top_p < 1.0:
sorted_indices = np.argsort(-probs)
sorted_probs = probs[sorted_indices]
cumulative_probs = np.cumsum(sorted_probs)
# 找到累计概率刚刚超过 top_p 的位置
cutoff_mask = cumulative_probs > top_p
# 保证至少保留第 1 个最高概率词
cutoff_mask[0] = False
# 剔除截断之外的词
filtered_indices = sorted_indices[cutoff_mask]
probs[filtered_indices] = 0.0
# 重新归一化
probs =
# 6. 多项分布随机采样抽签
return int(np.random.choice(len(probs), p=probs))
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: What is Min-P sampling, and what advantage does it have over Top-P?
Next: Speculative Decoding Verification & Accept/Reject →
💡

Core Mental Anchor / Mnemonic

Scale by temperature, keep top-K, truncate cumulative top-P mass (keeping >= 1 token), sample.

📐 Mathematical Derivation & Core Formula

Pi=ezi/T∑jezj/T,∑i∈TopPPi≥p,TopK:i≤KP_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}, \quad \sum_{i \in \mathrm{TopP}} P_i \ge p, \quad \mathrm{TopK}: i \le K
### Three-Stage Logit Modulation & Truncation
1. Temperature ($T$): Scales logits as z/Tz/T. As T→0T \to 0, Softmax sharpens into a Dirac delta / greedy argmax; T>1T > 1 flattens probabilities;
2. Top-$K$: Retains only the top KK logits and masks the rest to −∞-\infty;
3. Top-$P$ (Nucleus Sampling): Sorts probabilities descending and keeps the smallest prefix whose cumulative mass reaches PP (e.g., 0.90.9), adapting candidate count to context entropy.

🔄 Tensor Dimensions & Shape Flow

logits -> divide by T -> Top-K mask -inf -> Softmax -> cumulative Top-P mask & renorm -> sample token_id

🛡️ Numerical Stability & Pitfalls Checklist

  • Handle temperature == 0 directly via argmax rather than dividing by zero.
  • Always retain at least the highest-probability token in Top-P so remaining probabilities never sum to zero.

💻 Reference Code Implementation

import numpy as np

def sample_next_token(
    logits: np.ndarray,      # (vocab_size,) 未归一化预测
    temperature: float = 0.7,
    top_k: int = 50,
    top_p: float = 0.9
) -> int:
    # 1. 贪心特例
    if temperature == 0.0:
        return int(np.argmax(logits))
        
    # 2. 温度缩放
    scaled_logits = logits / temperature
    
    # 3. Top-K 截断
    if top_k > 0 and top_k < len(scaled_logits):
        # 寻找第 top_k 个最大值的阈值
        kth_val = np.partition(scaled_logits, -top_k)[-top_k]
        scaled_logits[scaled_logits < kth_val] = -np.inf
        
    # 4. Softmax 转换为有效概率分布
    max_l = np.max(scaled_logits)
    exp_l = np.exp(scaled_logits - max_l)
    probs = exp_l / np.sum(exp_l)
    
    # 5. Top-P (核采样) 截断
    if top_p < 1.0:
        sorted_indices = np.argsort(-probs)
        sorted_probs = probs[sorted_indices]
        cumulative_probs = np.cumsum(sorted_probs)
        
        # 找到累计概率刚刚超过 top_p 的位置
        cutoff_mask = cumulative_probs > top_p
        # 保证至少保留第 1 个最高概率词
        cutoff_mask[0] = False
        
        # 剔除截断之外的词
        filtered_indices = sorted_indices[cutoff_mask]
        probs[filtered_indices] = 0.0
        # 重新归一化
        probs = probs / np.sum(probs)
        
    # 6. 多项分布随机采样抽签
    return int(np.random.choice(len(probs), p=probs))

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
logits = np.array([10.0, 9.0, 1.0, 0.0])
# 贪心模式必定选第 0 个
assert sample_next_token(logits, temperature=0.0) == 0
# Top-K=1 等价于贪心
assert sample_next_token(logits, temperature=1.0, top_k=1) == 0
print("✓ 大模型生成采样器自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:What is Min-P sampling, and what advantage does it have over Top-P?
In low-entropy steps where one token clearly dominates, Top-P can still admit low-probability noise near the threshold tail. Min-P sets a relative floor pmin⁡=α⋅pmax⁡p_{\min} = \alpha \cdot p_{\max} (e.g., 0.05⋅pmax⁡0.05 \cdot p_{\max}), cleanly pruning tail noise when confident while preserving diversity when uncertain.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: INT8 Affine & Symmetric QuantizationBack to Part I & All 105Next: Speculative Decoding Verification & Accept/Reject →