Multimodal Reasoning & Agents

Multimodal Reasoning is the multi-step reasoning ability of VLMs on chart reasoning (ChartQA: reading axes, extracting values, tracking trends), spatial reasoning (direction/position/size/count), mathematics (MathVista) and scientific figures (ScienceQA). The core mechanism transfers text CoT (chain-of-thought) to visual contexts — first describe/locate visual evidence, then reason, with the final answer still produced by the autoregressive objective: $$P(y \mid v, q) = \prod_{t=1}^{L} P_\theta(y_t \mid v, q, y_{<t}), \quad y = [\text{CoT steps} \ldots \text{answer}]$$ The key difficulty of chart reasoning is numeric alignment (accurately reading ticks/bar heights, requiring high-resolution input or AnyRes-style upsampling); spatial reasoning is hard because location information gets compressed during patchification. Agent view: VLM acts as a perception layer with tool calling and video streams (single image → multi-image → video stream), converting visual state into executable instructions — the "eyes" of GUI/embodied/video agents.

interview questions on whether a VLM can reason, where it breaks, and how to improve it; multimodal agent selection (can the model read charts/reports, watch video streams, call tools); sanity-checking MathVista/ChartQA leaderboard claims.

early VLMs only perceived (description/recognition) without reasoning, performing near-chance on chart/math tasks; high-resolution/cropped inputs, interleaved pretraining (multi-image and long-context reasoning signals) and CoT instruction tuning make explicit reasoning chains emerge and transfer to agent scenarios. Failure modes concentrate on numeric misreading, evidence-reasoning decoupling, and spatial misjudgment — improvements target resolution, data, and decoding.

Multimodal CoT vs text CoT: differences and the core difficulty of chart reasoning?

VLM as agent perception: single-image, multi-image, and video-stream inputs?

Typical VLM failure modes on MathVista/ChartQA (numeric misreading, evidence decoupling)?

Why is spatial reasoning hard for VLMs and how is it improved?

Does multimodal reasoning come mainly from pretraining, SFT, or decoding? Evidence?