VLM Capabilities

VLM Capabilities is the unified set of autoregressive abilities of vision-language models: image captioning, visual QA (VQA), OCR (TextVQA/DocVQA), visual grounding (referring localization), multi-image comparison, chart and video understanding. Essentially every task is rewritten as "vision tokens + text prompt → text token sequence" autoregressive generation: $$p_\theta(y \mid v, q) = \prod_{t=1}^{L} p_\theta(y_t \mid v, q, y_{<t})$$ A single objective means one Masked-CE loss trains all abilities (task transfer + shared world knowledge). Grounding is the special case: the model outputs coordinate tokens, discretizing the continuous box $\hat{b} \in \mathbb{R}^4$ into grid coordinates (e.g. 0–1000 quantization) decoded alongside text tokens. Typical failures: text hallucination (fabricating semantic content from language prior) and OCR hallucination ("recognizing" text that does not match the image) — both rooted in language prior and data bias.

interview questions on what a VLM can do and how it is implemented uniformly; judging capability boundaries of new models (OCR ≠ counting; grounding ≠ fine-grained spatial reasoning); capability checklist for multimodal agent selection.

per-task specialist models underuse data and cannot share visual-linguistic knowledge; a unified autoregressive objective covers all abilities with one weight set and one loss, vision reasoning emerges after LLM unfreezing and SFT, and grounding/OCR directly inherit the LLM's instruction following and in-context learning.

Why do caption/VQA/OCR/grounding all reduce to autoregressive token generation? Write the unified objective.

How is visual grounding implemented? How are continuous boxes discretized into tokens?

Text hallucination vs OCR hallucination: difference, causes and examples?

VLM capability boundaries: what data and input representations enable OCR/chart/video?

Multi-image comparison vs video understanding: differences in input representation?