F6MediumPart F · Alignment & Reinforcement Learning

KL Divergence & Adaptive Penalty

⚡Flashcard Drill➔

Measure asymmetric divergence between probability distributions to regularize RLHF policies near reference.

⏱️ Time: O(C)
💾 Space: O(1)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def discrete_kl_divergence(p: np.ndarray, q: np.ndarray, eps: float = 1e-12) -> float:
"""
计算两离散概率分布的 KL(P || Q)
"""
p_safe =
q_safe = np.clip(q, eps, 1.0)
return float(np.sum(p_safe * np.log(p_safe / q_safe)))
def sample_level_kl_penalty(log_probs_actor: np.ndarray, log_probs_ref: np.ndarray) -> np.ndarray:
"""
RLHF 实际采样级别无偏近似: log(pi) - log(ref)
"""
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does Reverse KL often yield cleaner text than Forward KL when distilling large multimodal/multilingual teachers into smaller students?
Next: Multi-Armed Bandit: epsilon-Greedy & UCB1 →
💡

Core Mental Anchor / Mnemonic

Sum p * (log p - log q): Forward KL covers modes broadly; Reverse KL concentrates on dominant modes.

📐 Mathematical Derivation & Core Formula

DKL(P∥Q)=∑xP(x)log⁡P(x)Q(x)=Ex∼P[log⁡P(x)−log⁡Q(x)]D_{\mathrm{KL}}(P \parallel Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)} = \mathbb{E}_{x \sim P}[\log P(x) - \log Q(x)]
### Forward KL (Zero-Avoiding) vs. Reverse KL (Zero-Forcing)
KL divergence is asymmetric:
- Forward KL $D_{\mathrm{KL}}(P_{\text{data}} \parallel P_\theta)$: Penalizes Pθ(x)→0P_\theta(x) \to 0 wherever Pdata(x)>0P_{\text{data}}(x) > 0, encouraging broad mode-covering (mean-seeking);
- Reverse KL $D_{\mathrm{KL}}(P_\theta \parallel P_{\text{data}})$: Penalizes putting mass where Pdata(x)≈0P_{\text{data}}(x) \approx 0, encouraging mode-seeking concentration.
In RLHF, subtracting β(log⁡πθ−log⁡πref)\beta(\log \pi_\theta - \log \pi_{\mathrm{ref}}) penalizes drifting too far from the pretrained reference distribution.

🔄 Tensor Dimensions & Shape Flow

p, q: (C,) -> p * (log(p) - log(q)) -> sum -> scalar KL

🛡️ Numerical Stability & Pitfalls Checklist

  • Clamp probability inputs away from zero before taking logs to avoid log(0) = -inf.
  • Verify non-negativity (KL >= 0), hitting zero only when p equals q everywhere.

💻 Reference Code Implementation

import numpy as np

def discrete_kl_divergence(p: np.ndarray, q: np.ndarray, eps: float = 1e-12) -> float:
    """
    计算两离散概率分布的 KL(P || Q)
    """
    p_safe = np.clip(p, eps, 1.0)
    q_safe = np.clip(q, eps, 1.0)
    return float(np.sum(p_safe * np.log(p_safe / q_safe)))

def sample_level_kl_penalty(log_probs_actor: np.ndarray, log_probs_ref: np.ndarray) -> np.ndarray:
    """
    RLHF 实际采样级别无偏近似: log(pi) - log(ref)
    """
    return log_probs_actor - log_probs_ref

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
p = np.array([0.5, 0.5])
q = np.array([0.5, 0.5])
assert np.isclose(discrete_kl_divergence(p, q), 0.0)
q2 = np.array([0.9, 0.1])
assert discrete_kl_divergence(p, q2) > 0.0
print("✓ KL 散度计算自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:Why does Reverse KL often yield cleaner text than Forward KL when distilling large multimodal/multilingual teachers into smaller students?
A capacity-limited student forced to cover every teacher mode via Forward KL smears probability mass into low-density valleys between modes, generating incoherent text. Reverse KL lets the student concentrate faithfully on high-confidence modes.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Generalized Advantage Estimation (GAE)Back to Part F & All 105Next: Multi-Armed Bandit: epsilon-Greedy & UCB1 →