Dyna Architecture (Dyna-Q)

The Dyna architecture (Dyna-Q) hybridizes model learning with model-free learning: after each real interaction it updates Q on real experience, then treats the learned world model as a 'free environment' to generate $n$ imagined rollouts and applies the same Q-learning update to the mixture. Pseudocode essentials: ``` while True: a ← ε-greedy(s) # real experience: policy picks action r, s' ← env(s, a) # environment interaction Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') − Q(s,a)] # real update model(s,a) ← (r, s') # update the model repeat n times: # imagined rollouts mixed in s̃ ← random visited state; ã ← random action (r̃, s̃') ← model(s̃, ã) # simulate with the model Q(s̃,ã) ← Q(s̃,ã) + α[r̃ + γ·max Q(s̃',·) − Q(s̃,ã)] # imagined update ``` Real and imagined updates use the identical Q-learning rule; the model only expands the source of experience.

the canonical introduction to model-enhanced learning in tabular/simple environments; classic interview question 'simplest model-based paradigm' and 'first use of a model in RL'. Deep-RL counterpart: MBPO (Model-Based Policy Optimization) — model rollout data mixed into the replay buffer to train SAC/PPO.

slow convergence with scarce real experience; Dyna 'amplifies' each real sample into $n$ imagined samples, speeding Q-learning convergence markedly. Limitation: imagined states are sampled from the pool of previously visited states, so their distribution can mismatch states the policy actually reaches, amplifying model bias.

Test Your Knowledge: Practice Questions for "Dyna Architecture (Dyna-Q)"

Single choice pitfall questions with instant feedback and mistake tracking.