MC vs TD Learning

Monte Carlo (MC) updates from the full episode return, defined as discounted cumulative reward $G_t = \sum_{k=0}^{T-t-1} \gamma^k r_{t+k+1}$, where $T$ is the terminal time step, $r_{t+k+1}$ the reward $k+1$ steps after $t$, and $\gamma^k$ the discount; the update is $V(s_t) \leftarrow V(s_t) + \alpha\left[ G_t - V(s_t) \right]$.Since $G_t$ is an unbiased sample estimate of $v_\pi(s_t)$ (real samples only, no reliance on current estimates), MC is unbiased; but it sums $T-t$ stochastic rewards, so variance grows with episode length and $\gamma$, and learning must wait until episode end — episodic tasks only.

Temporal Difference (TD) learns from a one-step transition plus bootstrapping, with target $r_{t+1} + \gamma V(s_{t+1})$ — the real reward $r_{t+1}$ plus the discounted current estimate $V(s_{t+1})$; the update is $V(s_t) \leftarrow V(s_t) + \alpha\left[ r_{t+1} + \gamma V(s_{t+1}) - V(s_t) \right]$, the bracket being the TD error. Because the target contains an estimate, TD is biased (toward the current estimate), but it involves little stochasticity — low variance — learns at every step, and works on continuing tasks. MC corresponds to accumulating the TD error over a full trajectory ($\lambda = 1$); DP is full bootstrapping at $\lambda = 0$ — TD sits between the two; in tabular settings TD(0) converges to the true $v_\pi$ under standard conditions.

model-free online learning (unknown $P, R$) — Q-learning/DQN/Actor-Critic all build on TD errors; MC serves for diagnostic comparisons and episodic tasks (e.g. board evaluation updated per game). The bias-variance trade-off is a classic interview topic.

model-free environments rule out DP, while MC is impractical due to high variance and delayed updates; TD uses the one-step error as the learning signal via bootstrapping, trading bias (from the estimate) against variance (from randomness), achieving sample efficiency, online learning, and stability — the dominant learning paradigm.

Write $G_t$ and the TD target in full and explain each term. How do the two updates differ?

Why is MC unbiased with high variance, and TD biased with low variance? Where do the bias and variance come from?

What does TD's "bootstrapping" mean? Its key differences from MC?

How does TD(λ) unify MC and TD? What do λ = 0 and λ = 1 reduce to?

Why does TD dominate in practice? How does the TD error appear in DQN/Q-learning updates?