Multimodal DPO / RLHF-V

Multimodal DPO (MMDPO) extends DPO from text-only preference alignment to joint (image, text) inputs: under the same image $v$, a faithful answer $y_w$ (consistent with the image) vs a hallucinated answer $y_l$ (contradicting it) forms a preference pair, optimized directly via an implicit reward: $$\mathcal{L}_{DPO} = -\mathbb{E}_{(v,t,y_w,y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid v, t)}{\pi_{ref}(y_w \mid v, t)} - \beta \log \frac{\pi_\theta(y_l \mid v, t)}{\pi_{ref}(y_l \mid v, t)} \right) \right]$$ $t$ is the question text, $\pi_{ref}$ the frozen reference policy (SFT model), $\beta$ the reward scale (typically 0.1–0.5). The implicit reward $r_\theta(v, t, y) = \beta \log \frac{\pi_\theta(y \mid v,t)}{\pi_{ref}(y \mid v,t)}$ comes from the log-ratio of policy and reference — no reward model, no online sampling, no PPO; just maximum likelihood on offline preference pairs. POPE negative sampling (RLHF-V style): first use POPE probing to find objects the model frequently fabricates $\mathcal{O}_{\text{hall}}$ (candidates it answers Yes to though absent), then build hallucinated segments as negatives $y_l$ paired with faithful answers $y_w$; negatives come in random / popular / adversarial tiers, and adversarial hard negatives (the model's own most-hallucinated objects) align most effectively — targeted suppression of the model's actual failure modes rather than random noise.

offline preference alignment to suppress object hallucination and improve faithfulness (pairs built once, no online sampling cost); interview comparisons of why DPO over RLHF/GRPO for multimodal alignment; RLHF-V's hallucination-aware variant for hallucination-heavy scenarios.

naive text-DPO transfer to multimodal suffers from weak negatives (random negatives are obviously wrong and teach no boundary) and reward-model hallucination (a reward model can itself hallucinate and score hallucinated answers highly); POPE negative sampling targets the model's real defects (adversarial hard negatives substantially reduce hallucination), the implicit reward removes the reward model as a hallucination source, and unlike RLHF/GRPO (online sampling + verifier), MMDPO aligns offline with static image-text preference data — much cheaper and more stable.

Test Your Knowledge: Practice Questions for "Multimodal DPO / RLHF-V"

Single choice pitfall questions with instant feedback and mistake tracking.