World Model Learning

A world model parameterized by $\phi$ learns the environment's transition and reward distribution $P_\phi(s', r | s, a)$: given state $s$ and action $a$, predict next state $s'$ and immediate reward $r$. Training is maximum likelihood estimation (MLE): $$\max_\phi \; \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}} \left[ \log P_\phi(s', r | s, a) \right]$$ with $\mathcal{D}$ the collected real interaction dataset. Once learned, the model substitutes for the environment to generate 'imagined rollouts', reusing each real sample for many simulated updates and plans.

environments where real interaction is expensive or dangerous (robotics, autonomous driving, healthcare); pixel inputs needing latent-space models (cf. Dreamer/MuZero); imagined planning with a planner (cf. MPC). Common interview entry: model-based vs model-free.

model-free RL needs vast real interaction, hurting sample efficiency; a world model 'internalizes' the environment as a first-order approximation so the agent trains and plans repeatedly in imagination — roughly an order-of-magnitude sample-efficiency gain — at the cost of model bias polluting the policy (compounding error $O(H^2\varepsilon)$, see model-vs-model-free).

What is the world model's training objective? Write the MLE objective and explain each term.

Why does a world model improve sample efficiency? Imagined rollouts vs real interaction?

How does model error hurt the downstream policy? What is compounding error O(H²ε)?

Explicit (pixel/state-level) vs implicit (latent) world models — differences? One example each?

Beyond classic RL, what new world-model applications exist for LLMs/agents?