F5HardPart F · Alignment & Reinforcement Learning
Generalized Advantage Estimation (GAE)
Balance TD bias and Monte Carlo variance via backward exponential discounting of TD residuals.
⏱️ Time:
O(T) 线性逆向遍历💾 Space:
O(T)🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_gae(
rewards: np.ndarray, # (T,) 时间步奖励
values: np.ndarray, # (T+1,) 包含终止状态预估的价值
gamma: float = 0.99,
lam: float = 0.95
) -> np.ndarray:
T = len(rewards)
advantages = np.zeros(T, dtype=np.float64)
last_gae = 0.0
# 从 T-1 逆序递归计算至 0
for t in reversed(range(T)):
# 1. 计算当前步的时序差分误差 delta
delta =
# 2. 递归更新 GAE
advantages[t] =
last_gae = advantages[t]
return advantages
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How is the Critic value target computed alongside GAE in PPO training?
💡
Core Mental Anchor / Mnemonic
Compute TD residuals delta_t, then accumulate backward with decay factor gamma * lambda.
📐 Mathematical Derivation & Core Formula
### Bias-Variance Interpolation via Exponential Trace
1-step TD error has low variance but high critic bias; full Monte Carlo returns are unbiased but high variance.
GAE (Schulman et al., 2015) takes an exponentially weighted sum of -step TD residuals backwards in time:
Setting recovers 1-step TD; recovers full MC advantage; typical values are .
1-step TD error has low variance but high critic bias; full Monte Carlo returns are unbiased but high variance.
GAE (Schulman et al., 2015) takes an exponentially weighted sum of -step TD residuals backwards in time:
Setting recovers 1-step TD; recovers full MC advantage; typical values are .
🔄 Tensor Dimensions & Shape Flow
rewards: (T,), values: (T+1,) -> reverse loop -> delta_t -> adv_t = delta + gamma*lam*adv_{t+1} -> (T,)
🛡️ Numerical Stability & Pitfalls Checklist
- Iterate strictly backward from T-1 down to 0 so adv_{t+1} propagates correctly.
- Provide T+1 value estimates where terminal value V_T is zeroed on episode termination (
done).
💻 Reference Code Implementation
import numpy as np
def compute_gae(
rewards: np.ndarray, # (T,) 时间步奖励
values: np.ndarray, # (T+1,) 包含终止状态预估的价值
gamma: float = 0.99,
lam: float = 0.95
) -> np.ndarray:
T = len(rewards)
advantages = np.zeros(T, dtype=np.float64)
last_gae = 0.0
# 从 T-1 逆序递归计算至 0
for t in reversed(range(T)):
# 1. 计算当前步的时序差分误差 delta
delta = rewards[t] + gamma * values[t + 1] - values[t]
# 2. 递归更新 GAE
advantages[t] = delta + gamma * lam * last_gae
last_gae = advantages[t]
return advantages
🧪 Runnable Assertions & Validation
Copy and run directly in Python / Jupyter:
import numpy as np
rewards = np.array([1.0, 1.0, 1.0])
values = np.array([0.5, 0.5, 0.5, 0.0])
adv = compute_gae(rewards, values, gamma=0.99, lam=0.95)
assert len(adv) == 3
assert adv[0] > 0, "正向奖励应产生正优势"
print("✓ GAE 广义优势估计自测通过")🎯 Core Follow-up Inquiries & Reference Answers
Q1:How is the Critic value target computed alongside GAE in PPO training?
The value target is formed directly as , and the value head minimizes squared error .
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.