Dreamer RSSM (Latent Imagination)

Dreamer learns a world model from pixel observations with an RSSM (Recurrent State-Space Model) and trains the policy in latent imagination. The RSSM latent state has two paths: ① deterministic path: $h_t = f(h_{t-1}, z_{t-1}, a_{t-1})$ — a recurrent network integrating history;② stochastic path: $z_t \sim q(z_t | h_t, o_t)$ — a stochastic latent variable encoding the current observation; reconstruction heads: $\hat{o}_t = o(h_t, z_t)$ (reconstruct observation) and $\hat{r}_t = r(h_t, z_t)$ (predict reward).

Policy training in latent imagination: two phases — first train the world model on real sequences only (observation reconstruction + reward prediction + KL regularizer keeping $q$ close to the prior); then freeze the model and 'dream': from the current latent state, roll out the latent trajectory with the current policy, then update actor-critic to maximize discounted return along this purely imagined trajectory. Policy and value heads are trained entirely in latent space, never touching the real environment.

pixel-based control (DMC, Atari), robot learning; Dreamer V1/V2/V3 are the SOTA model-based line for pixel control; interviews probe how 'dreaming in latent space' saves samples.

pixel-level model-free RL is extremely sample-hungry; the RSSM's deterministic + stochastic hybrid captures both temporal dependencies and observation uncertainty, keeping long imagined rollouts stable (pure stochastic models diverge in imagination) — roughly an order-of-magnitude sample-efficiency gain over model-free SOTA.

What do RSSM's deterministic and stochastic states capture? Write and explain h_t = f(h_{t-1}, z_{t-1}, a_{t-1}).

How does Dreamer train its policy 'in dreams'? The full latent imagined rollout procedure?

Why is the stochastic latent z_t necessary for imagined rollouts? What happens with purely deterministic or purely stochastic models?

Dreamer vs MuZero latent modeling: similarities and differences?

How much more sample-efficient is Dreamer than model-free SOTA on pixel control, and why?