F5HardPart F · Alignment & Reinforcement Learning

Generalized Advantage Estimation (GAE)

⚡Flashcard Drill➔

Balance TD bias and Monte Carlo variance via backward exponential discounting of TD residuals.

⏱️ Time: O(T) 线性逆向遍历
💾 Space: O(T)
🎯 Interactive Self-Practice Loop (Attempt Before Peeking)
Practice in 4 steps: recall mechanism, complete core lines, check edge cases, and log mastery locally.
Step 1 · Mechanism Recall & Stability Self-Check
Step 2 · Complete Key Implementation Lines
import numpy as np
def compute_gae(
rewards: np.ndarray, # (T,) 时间步奖励
values: np.ndarray, # (T+1,) 包含终止状态预估的价值
gamma: float = 0.99,
lam: float = 0.95
) -> np.ndarray:
T = len(rewards)
advantages = np.zeros(T, dtype=np.float64)
last_gae = 0.0
# 从 T-1 逆序递归计算至 0
for t in reversed(range(T)):
# 1. 计算当前步的时序差分误差 delta
delta =
# 2. 递归更新 GAE
advantages[t] =
last_gae = advantages[t]
return advantages
Step 3 · Verify Boundary Cases & Assertions
Step 4 · Follow-Up Drill & Review Outcome
Q1: How is the Critic value target computed alongside GAE in PPO training?
Next: KL Divergence & Adaptive Penalty →
💡

Core Mental Anchor / Mnemonic

Compute TD residuals delta_t, then accumulate backward with decay factor gamma * lambda.

📐 Mathematical Derivation & Core Formula

A^tGAE=∑l=0∞(γλ)lδt+lV,δtV=rt+γV(st+1)−V(st)\hat{A}_t^{\mathrm{GAE}} = \sum_{l=0}^\infty (\gamma \lambda)^l \delta_{t+l}^V, \quad \delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t)
### Bias-Variance Interpolation via Exponential Trace
1-step TD error δtV=rt+γV(st+1)−V(st)\delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t) has low variance but high critic bias; full Monte Carlo returns are unbiased but high variance.
GAE (Schulman et al., 2015) takes an exponentially weighted sum of kk-step TD residuals backwards in time:
A^t=δtV+(γλ)A^t+1\hat{A}_t = \delta_t^V + (\gamma \lambda) \hat{A}_{t+1}
Setting λ=0\lambda = 0 recovers 1-step TD; λ=1\lambda = 1 recovers full MC advantage; typical values are γ=0.99,λ=0.95\gamma = 0.99, \lambda = 0.95.

🔄 Tensor Dimensions & Shape Flow

rewards: (T,), values: (T+1,) -> reverse loop -> delta_t -> adv_t = delta + gamma*lam*adv_{t+1} -> (T,)

🛡️ Numerical Stability & Pitfalls Checklist

  • Iterate strictly backward from T-1 down to 0 so adv_{t+1} propagates correctly.
  • Provide T+1 value estimates where terminal value V_T is zeroed on episode termination (done).

💻 Reference Code Implementation

import numpy as np

def compute_gae(
    rewards: np.ndarray,      # (T,) 时间步奖励
    values: np.ndarray,       # (T+1,) 包含终止状态预估的价值
    gamma: float = 0.99,
    lam: float = 0.95
) -> np.ndarray:
    T = len(rewards)
    advantages = np.zeros(T, dtype=np.float64)
    last_gae = 0.0
    
    # 从 T-1 逆序递归计算至 0
    for t in reversed(range(T)):
        # 1. 计算当前步的时序差分误差 delta
        delta = rewards[t] + gamma * values[t + 1] - values[t]
        # 2. 递归更新 GAE
        advantages[t] = delta + gamma * lam * last_gae
        last_gae = advantages[t]
        
    return advantages

🧪 Runnable Assertions & Validation

Copy and run directly in Python / Jupyter:

import numpy as np
rewards = np.array([1.0, 1.0, 1.0])
values = np.array([0.5, 0.5, 0.5, 0.0])
adv = compute_gae(rewards, values, gamma=0.99, lam=0.95)
assert len(adv) == 3
assert adv[0] > 0, "正向奖励应产生正优势"
print("✓ GAE 广义优势估计自测通过")

🎯 Core Follow-up Inquiries & Reference Answers

Q1:How is the Critic value target computed alongside GAE in PPO training?
The value target is formed directly as Vtarget=A^GAE+VoldV_{\text{target}} = \hat{A}^{\text{GAE}} + V_{\text{old}}, and the value head minimizes squared error 12(Vθ(st)−Vtarget)2\frac{1}{2}(V_\theta(s_t) - V_{\text{target}})^2.
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
← Prev: Group Relative Policy Optimization (GRPO)Back to Part F & All 105Next: KL Divergence & Adaptive Penalty →