F6MediumPart F · Alignment & Reinforcement Learning
KL Divergence & Adaptive Penalty
Measure asymmetric divergence between probability distributions to regularize RLHF policies near reference.
⏱️ Time:
O(C)💾 Space:
O(1)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def discrete_kl_divergence(p: np.ndarray, q: np.ndarray, eps: float = 1e-12) -> float:
"""
计算两离散概率分布的 KL(P || Q)
"""
p_safe =
q_safe = np.clip(q, eps, 1.0)
return float(np.sum(p_safe * np.log(p_safe / q_safe)))
def sample_level_kl_penalty(log_probs_actor: np.ndarray, log_probs_ref: np.ndarray) -> np.ndarray:
"""
RLHF 实际采样级别无偏近似: log(pi) - log(ref)
"""
return
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: Why does Reverse KL often yield cleaner text than Forward KL when distilling large multimodal/multilingual teachers into smaller students?
💡
Core Mental Anchor / Mnemonic
Sum p * (log p - log q): Forward KL covers modes broadly; Reverse KL concentrates on dominant modes.
📐 Mathematical Derivation & Core Formula
### Forward KL (Zero-Avoiding) vs. Reverse KL (Zero-Forcing)
KL divergence is asymmetric:
- Forward KL $D_{\mathrm{KL}}(P_{\text{data}} \parallel P_\theta)$: Penalizes wherever , encouraging broad mode-covering (mean-seeking);
- Reverse KL $D_{\mathrm{KL}}(P_\theta \parallel P_{\text{data}})$: Penalizes putting mass where , encouraging mode-seeking concentration.
In RLHF, subtracting penalizes drifting too far from the pretrained reference distribution.
KL divergence is asymmetric:
- Forward KL $D_{\mathrm{KL}}(P_{\text{data}} \parallel P_\theta)$: Penalizes wherever , encouraging broad mode-covering (mean-seeking);
- Reverse KL $D_{\mathrm{KL}}(P_\theta \parallel P_{\text{data}})$: Penalizes putting mass where , encouraging mode-seeking concentration.
In RLHF, subtracting penalizes drifting too far from the pretrained reference distribution.
🔄 Tensor Dimensions & Shape Flow
p, q: (C,) -> p * (log(p) - log(q)) -> sum -> scalar KL
🛡️ Numerical Stability & Pitfalls Checklist
- Clamp probability inputs away from zero before taking logs to avoid
log(0) = -inf. - Verify non-negativity (
KL >= 0), hitting zero only when p equals q everywhere.
💻 Reference Code Implementation
import numpy as np
def discrete_kl_divergence(p: np.ndarray, q: np.ndarray, eps: float = 1e-12) -> float:
"""
计算两离散概率分布的 KL(P || Q)
"""
p_safe = np.clip(p, eps, 1.0)
q_safe = np.clip(q, eps, 1.0)
return float(np.sum(p_safe * np.log(p_safe / q_safe)))
def sample_level_kl_penalty(log_probs_actor: np.ndarray, log_probs_ref: np.ndarray) -> np.ndarray:
"""
RLHF 实际采样级别无偏近似: log(pi) - log(ref)
"""
return log_probs_actor - log_probs_ref
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
p = np.array([0.5, 0.5])
q = np.array([0.5, 0.5])
assert np.isclose(discrete_kl_divergence(p, q), 0.0)
q2 = np.array([0.9, 0.1])
assert discrete_kl_divergence(p, q2) > 0.0
print("✓ KL 散度计算自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:Why does Reverse KL often yield cleaner text than Forward KL when distilling large multimodal/multilingual teachers into smaller students?
A capacity-limited student forced to cover every teacher mode via Forward KL smears probability mass into low-density valleys between modes, generating incoherent text. Reverse KL lets the student concentrate faithfully on high-confidence modes.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.