GAIL & Inverse RL

Inverse RL (IRL) recovers a reward function from expert trajectories; GAIL (Generative Adversarial Imitation Learning) merges adversarial training with max-entropy IRL to learn a policy directly: $\min_\pi \max_D \mathbb{E}_{\pi_E}[\log D(s,a)] + \mathbb{E}_\pi[\log(1-D(s,a))] - \lambda \mathcal{H}(\pi)$ — discriminator $D$ tells expert $(s,a)$ pairs from the policy's, and the policy fools it into matching the expert distribution, with entropy $\mathcal{H}(\pi)$ preventing collapse.

rewards are hard to design but expert demos are plentiful — robot teaching, driving, games; the learned implicit reward also enables behavior explanation and transfer across tasks.

upgrades imitation from 'passively fitting expert actions' (BC) to 'actively matching the expert trajectory distribution' — the discriminator supplies dense adversarial feedback at every step, so errors never compound quadratically as in BC; meanwhile max-entropy IRL assumes expert trajectories satisfy $p(\tau) \propto \exp(R(\tau))$ — higher-reward trajectories are exponentially more likely to be chosen — giving a principled solution space for the reward and ruling out trivial solutions (e.g., constant reward) and IRL's ill-posedness.

Write the GAIL objective and explain each term; which divergence between the expert and policy distributions does it minimize?

Why does GAIL mitigate compounding error compared to BC? What signal does the discriminator provide?

What is the max-entropy IRL assumption $p(\tau) \propto \exp(R(\tau))$? Which IRL ill-posedness (e.g., constant-reward trivial solution) does it fix?

Common issues with GAIL's discriminator (implicit reward)? Role of the entropy bonus $-\lambda \mathcal{H}(\pi)$?

IRL/GAIL and RLHF reward modeling both infer preferences from behavior — what are the key differences?