Dyna Architecture (Dyna-Q)

The Dyna architecture (Dyna-Q) hybridizes model learning with model-free learning: after each real interaction it updates Q on real experience, then treats the learned world model as a 'free environment' to generate $n$ imagined rollouts and applies the same Q-learning update to the mixture. Pseudocode essentials: ``` while True: a ← ε-greedy(s) # real experience: policy picks action r, s' ← env(s, a) # environment interaction Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') − Q(s,a)] # real update model(s,a) ← (r, s') # update the model repeat n times: # imagined rollouts mixed in s̃ ← random visited state; ã ← random action (r̃, s̃') ← model(s̃, ã) # simulate with the model Q(s̃,ã) ← Q(s̃,ã) + α[r̃ + γ·max Q(s̃',·) − Q(s̃,ã)] # imagined update ``` Real and imagined updates use the identical Q-learning rule; the model only expands the source of experience.

the canonical introduction to model-enhanced learning in tabular/simple environments; classic interview question 'simplest model-based paradigm' and 'first use of a model in RL'. Deep-RL counterpart: MBPO (Model-Based Policy Optimization) — model rollout data mixed into the replay buffer to train SAC/PPO.

slow convergence with scarce real experience; Dyna 'amplifies' each real sample into $n$ imagined samples, speeding Q-learning convergence markedly. Limitation: imagined states are sampled from the pool of previously visited states, so their distribution can mismatch states the policy actually reaches, amplifying model bias.

What is the core idea of Dyna-Q? Write the key pseudocode steps.

How are states/actions sampled for imagined rollouts? Problems if the imagined distribution mismatches reality?

How does n (imagined steps per interaction) affect convergence speed and compute?

Dyna vs Priority Sweeping: what differs?

What is Dyna's deep-RL counterpart (e.g. how MBPO mixes model rollouts)?