High-Res & Dynamic Tokens

High-resolution input is the key to VLM performance on OCR, documents and small objects; two mainstream schemes: ① LLaVA-1.5/1.6 AnyRes — split the high-res image into an $n_h \times n_w$ grid of 336×336 cells, run each cell through the vision encoder for 576 tokens (ViT-14), and prepend a global thumbnail to preserve overall semantics; total tokens $N_{\text{tokens}} \approx (n_h \times n_w + 1) \times 576$; ② Qwen2-VL dynamic resolution — vision tokens are generated according to the native aspect ratio within the pixel range $[256 \times 28 \times 28,\ 1280 \times 28 \times 28]$ (roughly 1 token per $28 \times 28$ pixels), i.e. 256 minimum and 1280 maximum tokens, with token count proportional to image area instead of fixed grids.

document understanding, OCR, screenshot QA, fine-grained visual grounding; interviews ask about the difference between AnyRes and dynamic tokens, and why high resolution boosts OCR.

resizing to 336×336 destroys small text/objects, while feeding full-resolution pixels explodes the token budget (a 4K image would cost tens of thousands of tokens, blowing the LLM context); grid patching controls cost without losing resolution — LLaVA-1.6 offers 1/2/4-grid tiers (about 672/1296/2304 tokens), and Qwen2-VL makes token count vary continuously with area, trading detail fidelity against context length in a configurable way; the resolution gain delivers double-digit improvements on OCR/document benchmarks (LLaVA-1.6 clearly beats 1.5 on TextVQA and similar).

How does AnyRes work? How are grids split and what does the global thumbnail do?

LLaVA-1.6 tiers (1/2/4 grids): how many visual tokens does each produce?

Qwen2-VL dynamic resolution vs AnyRes fixed grids: differences?

Why does high resolution drastically improve OCR/document ability? At what cost?

How does visual token count affect LLM attention cost (quadratic in sequence length)?