Vision Encoder

The vision encoder of a VLM usually reuses CLIP's ViT (CLIP-ViT-L/14 is the LLaVA-1.5 default) or SigLIP (the LLaVA-1.6 upgrade): the image is split into $p \times p$ patches, position-encoded, and passed through a Transformer, with token count $T = (H/p) \times (W/p)$ — e.g. a 336×336 input with 14×14 patches yields $24 \times 24 = 576$ visual tokens. SigLIP replaces the softmax contrastive loss with a sigmoid loss, removing the dependence on in-batch negatives and temperature, so training uses less memory and smaller batches.

the first stage of virtually every modular VLM (LLaVA/Qwen-VL/InternVL); interviews probe “why freeze the vision tower” and the patch-size vs resolution trade-off; high-resolution needs pair it with AnyRes or dynamic-token schemes.

reusing a pretrained CLIP vision tower gives zero-shot alignment for free, and freezing it in early training stages (training only the projector) prevents visual gradients from corrupting text embeddings while saving roughly half the vision tower's memory and compute; the downside is that fixed patches lose fine details — at 336×336 with ViT-14 you only get 576 tokens, so small text/objects are under-encoded, which is why LLaVA-1.6 uses AnyRes grid patching (576 tokens per grid) and Qwen2-VL uses dynamic resolution: trading more tokens for high-res detail.

How many visual tokens does 336×336 + ViT-14 produce? Give the formula.

Why freeze the vision encoder in early VLM stages? When is it unfrozen?

CLIP ViT vs SigLIP: what does the sigmoid loss fix?

Is a larger vision ViT always better? How to weigh training cost vs gain?

With a frozen vision tower, why can high-resolution ability still emerge (AnyRes patching)?