Model Predictive Control (MPC)

MPC (Model Predictive Control) uses a known or learned environment model to plan over a finite horizon $H$ at every decision step, maximizing cumulative return: $$J = \sum_{k=0}^{H-1} \gamma^k \, r(s_k, a_k)$$with $s_{k+1} \sim P_\phi(\cdot | s_k, a_k)$ rolled out by the model. The planner searches an action sequence $(a_0, \dots, a_{H-1})$ from the current state $s_0$, but executes only the first action and re-plans at the next step — a receding horizon that continuously re-anchors to real observations, resisting model error and disturbances.

Action search: ① Random Shooting — sample $K$ full action sequences uniformly from the prior, evaluate by $J$, keep the best; simple but sample-inefficient. ② CEM (Cross-Entropy Method) — iterate: sample $K$ sequences → keep the top-$\rho$ elites by $J$ → fit a Gaussian to the elites (update mean/variance) → resample; after several iterations, act on the center action of the final distribution. CEM typically finds better solutions with fewer samples than Random Shooting.

continuous control (robotics, autonomous driving, drones), tasks with a cheap simulator or known dynamics; MPC trains no policy — 'planning is the policy' — and can swap reward objectives on the fly.

avoids the huge sampling cost of learning a policy by searching optimal actions online; limits are model error and finite horizon $H$ — too small $H$ sees only local gains, too large compounds error, and per-step planning is computationally expensive.

Test Your Knowledge: Practice Questions for "Model Predictive Control (MPC)"

Single choice pitfall questions with instant feedback and mistake tracking.