Model-Based vs Model-Free

The two RL paradigms contrasted:

| Aspect | Model-Based | Model-Free | | --- | --- | --- | | Sample efficiency | High (imagined rollouts reuse data) | Low (needs vast real interaction) | | Model bias | Yes (a wrong model misleads the policy) | None (learns from real interaction) | | Policy expression | Planning is the policy; goals swap on the fly | Fixed policy; retrain for new goals | | Compute | Training + per-step planning cost | Cheap inference, low deployment cost | | Examples | Dyna, MPC, MuZero, Dreamer | DQN, PPO, SAC, GRPO |

Core risk — compounding error: with per-step model error $\varepsilon$ (e.g. TV distance $D_{\mathrm{TV}}(P_\phi, P^*) \le \varepsilon$), the cumulative error of an $H$-step model rollout is $O(H^2 \varepsilon)$. Intuition: the error $\varepsilon$ introduced at step $k$ is amplified ~$H-k$ times through the remaining value updates; summing over $k$: $\sum_{k=1}^{H} \varepsilon (H-k) = \varepsilon \cdot \frac{H(H-1)}{2} = O(H^2 \varepsilon)$ — error grows quadratically with horizon.

a must-ask comparison in interviews (sample efficiency vs model bias); algorithm choice — pick MF when interaction is cheap/data is abundant (stable, simple), pick MB when interaction is expensive or the goal is reusable; offline RL often combines both (model-generated data + conservative Q constraints).

the two are complementary — MF is unbiased but sample-hungry, MB is efficient but biased; hybrid routes (Dyna-style replay, MBPO mixing model rollouts into the buffer, offline RL augmenting datasets with the model) get the best of both.

Core differences between model-based and model-free: sample efficiency, model bias, compute?

Derive the compounding error O(H²ε): give the intuition and the summation.

When to choose model-based vs model-free (interaction cost, goal variability, data scale)?

How to hybridize the two to mitigate each other's flaws (Dyna, MBPO, offline RL)?

How to quantify/detect model bias? How to plan robustly under model uncertainty?