Reward Design & Hacking

Reward design comes in three forms: sparse rewards ($\pm 1$ only at termination — unbiased but hard to explore), dense rewards (signal every step — faster learning but potentially misleading), and potential-based reward shaping — augmenting the reward with $$F(s, a, s') = \gamma\Phi(s') - \Phi(s)$$ for any potential function $\Phi(s)$ (e.g. negative distance to goal) and discount $\gamma$. Ng et al. (1999) theorem: potential-based shaping does not change the optimal policy — along any trajectory $\sum_t F_t = \gamma\Phi(s_T) - \Phi(s_0)$ telescopes to boundary terms only, so value ordering is preserved and domain knowledge accelerates learning without misdirection.

Reward hacking: the policy finds and exploits loopholes in the reward function, scoring high without actually solving the task — e.g. an LLM bypassing unit tests, sycophantic 'I am very honest' text that inflates reward-model scores, or a game agent stalling in a corner. Mitigations: verifiable rewards (RLVR — the verifier has no exploitable loopholes), KL constraints limiting drift, adversarial training/regularization of reward models, multi-sample verification and human auditing.

reward-function design for games, robotics, and LLM alignment; interviews probe the shaping theorem derivation and LLM hacking case studies.

sparse rewards stall learning; poorly designed dense rewards get exploited and cause capability regression — good reward design balances learning speed (shaping) against hackability (verifiability).

Pros and cons of sparse vs dense rewards? How to choose per task?

Write the shaping formula F = γΦ(s') − Φ(s) and explain why it preserves the optimal policy (Ng's theorem)?

What is reward hacking? Give an LLM-scenario example (RLHF/RLVR)?

How to mitigate reward hacking? Why are verifiable rewards immune?

Best practices for reward design in LLM reasoning RL (verifiable rewards vs reward models vs PRMs)?