Multimodal Hallucination

Multimodal Hallucination is VLM output that contradicts the image, mainly two kinds: object hallucination (mentioning objects not present) and attribute hallucination (wrong color/count/position). Three driving forces: (1) text prior dominance: when generating, visual evidence is weaker than the LLM's text statistics, $p_\theta(y_t \mid x_{<t}, v)$ leans on $v$ too little and the model fills in from language habits; (2) data co-occurrence bias: objects frequently co-occur in captions (e.g. person and surfboard in COCO), so the model learns co-occurrence statistics, not "is it in the image"; (3) training objective and decoding amplification: Masked CE supervises only text tokens, vision tokens get no gradient supervision; greedy/sampling decoding amplifies high-entropy hallucinated tokens. Decoding-level mitigation: Visual Contrastive Decoding (VCD) cancels the language prior with a distorted image $v'$ (e.g. Gaussian blur): $$q(y_t) = \text{softmax}\left( (1+\alpha)\, \log p_\theta(y_t \mid x_{<t}, v) - \alpha\, \log p_\theta(y_t \mid x_{<t}, v') \right)$$ $\alpha$ controls prior-cancellation strength (typically 0.1–1); OPERA penalizes oversampling behavior where decoding repeats without new attention. Evaluation uses POPE probing and CHAIR word-level metrics: $CHAIR_S = \frac{|\text{hallucinated objects}|}{|\text{all mentioned objects}|}$.

a must-ask interview topic (causes and mitigations); hallucination audit before deployment in fact-sensitive domains (medical/document); preference-data construction in alignment (see the multimodal-dpo card).

hallucination makes VLMs unusable in fact-sensitive scenarios and cannot be cured by scale alone (stronger priors hallucinate more); systematically mitigated at three levels — data debiasing (balanced negatives/hard negatives), alignment (MMDPO/RLHF-V with POPE negatives), decoding intervention (VCD/OPERA) — with POPE/CHAIR providing quantifiable evaluation; the levels are complementary: data fixes the prior, alignment fixes preferences, decoding fixes runtime behavior.

Test Your Knowledge: Practice Questions for "Multimodal Hallucination"

Single choice pitfall questions with instant feedback and mistake tracking.