World Model Learning

A world model parameterized by $\phi$ learns the environment's transition and reward distribution $P_\phi(s', r | s, a)$: given state $s$ and action $a$, predict next state $s'$ and immediate reward $r$. Training is maximum likelihood estimation (MLE): $$\max_\phi \; \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}} \left[ \log P_\phi(s', r | s, a) \right]$$ with $\mathcal{D}$ the collected real interaction dataset. Once learned, the model substitutes for the environment to generate 'imagined rollouts', reusing each real sample for many simulated updates and plans.

environments where real interaction is expensive or dangerous (robotics, autonomous driving, healthcare); pixel inputs needing latent-space models (cf. Dreamer/MuZero); imagined planning with a planner (cf. MPC). Common interview entry: model-based vs model-free.

model-free RL needs vast real interaction, hurting sample efficiency; a world model 'internalizes' the environment as a first-order approximation so the agent trains and plans repeatedly in imagination — roughly an order-of-magnitude sample-efficiency gain — at the cost of model bias polluting the policy (compounding error $O(H^2\varepsilon)$, see model-vs-model-free).

Test Your Knowledge: Practice Questions for "World Model Learning"

Single choice pitfall questions with instant feedback and mistake tracking.