Multimodal Eval Benchmarks

Multimodal benchmarks are specialized by dimension: MMBench for general ability, MMMU for academic reasoning, POPE for hallucination — each with its own question format and metrics. MMBench: 30 abilities across 16 dimensions (~3000 MCQs) covering perception/cognition/knowledge, with a strict "output only the option letter" format to avoid open-generation parsing ambiguity; the metric is MCQ accuracy $Acc = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}[\hat{y}_i = y_i]$; MMMU: college-level multidisciplinary questions (30+ subjects across arts/business/science/engineering, 11.5K items) designed so that "visual information is necessary" (no-image performance ≈ 25% chance), testing perception + knowledge + reasoning jointly; POPE (Polling-based Object Probing Evaluation): object-presence probing — asks "Is there a [object] in the image?" for each candidate object; positives are objects truly present, negatives are absent objects in three difficulty tiers (random / popular / adversarial); metrics are binary precision/recall/F1: $$F_1 = \frac{2 \cdot P \cdot R}{P + R}, \quad P = \frac{TP}{TP+FP}, \quad R = \frac{TP}{TP+FN}$$ plus the yes-ratio (fraction of Yes answers) capturing overall hallucination tendency. Others in common use: ChartQA (numeric reasoning), TextVQA/DocVQA (OCR), ScienceQA (scientific figures), HallusionBench/MMHal-Bench (hallucination + reasoning, the latter scored by LLM-as-judge over multiple dimensions).

interview questions on which benchmark evaluates what and why; model selection cross-checked via ability matrix (MMBench) + hallucination risk (POPE) + hard reasoning (MMMU); regression before releases.

a single large VQA benchmark cannot attribute ability — high scores may come from language priors or retrieval shortcuts (e.g. answering without looking at the image); the tripartite design — dimension-wise matrix (MMBench), vision-necessary items (MMMU), adversarial presence probing (POPE) — respectively closes the gaps of non-answers, language shortcuts, and object hallucination, making scores attributable to concrete capability holes.

Why is MMBench's ability taxonomy and single-letter MCQ design effective?

MMMU vs generic VQA benchmarks; why is visual information designed to be necessary?

POPE's three negative-construction strategies (random/popular/adversarial)?

Write POPE's Precision/Recall/F1 and yes-ratio; what does each measure?

Methodological differences among CHAIR / MMHal-Bench / HallusionBench?