MC vs TD Learning

Monte Carlo (MC) updates from the full episode return, defined as discounted cumulative reward $G_t = \sum_{k=0}^{T-t-1} \gamma^k r_{t+k+1}$, where $T$ is the terminal time step, $r_{t+k+1}$ the reward $k+1$ steps after $t$, and $\gamma^k$ the discount; the update is $V(s_t) \leftarrow V(s_t) + \alpha\left[ G_t - V(s_t) \right]$.Since $G_t$ is an unbiased sample estimate of $v_\pi(s_t)$ (real samples only, no reliance on current estimates), MC is unbiased; but it sums $T-t$ stochastic rewards, so variance grows with episode length and $\gamma$, and learning must wait until episode end — episodic tasks only.

Temporal Difference (TD) learns from a one-step transition plus bootstrapping, with target $r_{t+1} + \gamma V(s_{t+1})$ — the real reward $r_{t+1}$ plus the discounted current estimate $V(s_{t+1})$; the update is $V(s_t) \leftarrow V(s_t) + \alpha\left[ r_{t+1} + \gamma V(s_{t+1}) - V(s_t) \right]$, the bracket being the TD error. Because the target contains an estimate, TD is biased (toward the current estimate), but it involves little stochasticity — low variance — learns at every step, and works on continuing tasks. MC corresponds to accumulating the TD error over a full trajectory ($\lambda = 1$); DP is full bootstrapping at $\lambda = 0$ — TD sits between the two; in tabular settings TD(0) converges to the true $v_\pi$ under standard conditions.

model-free online learning (unknown $P, R$) — Q-learning/DQN/Actor-Critic all build on TD errors; MC serves for diagnostic comparisons and episodic tasks (e.g. board evaluation updated per game). The bias-variance trade-off is a classic interview topic.

model-free environments rule out DP, while MC is impractical due to high variance and delayed updates; TD uses the one-step error as the learning signal via bootstrapping, trading bias (from the estimate) against variance (from randomness), achieving sample efficiency, online learning, and stability — the dominant learning paradigm.

Test Your Knowledge: Practice Questions for "MC vs TD Learning"

Single choice pitfall questions with instant feedback and mistake tracking.