Vision Encoder

The vision encoder of a VLM usually reuses CLIP's ViT (CLIP-ViT-L/14 is the LLaVA-1.5 default) or SigLIP (the LLaVA-1.6 upgrade): the image is split into $p \times p$ patches, position-encoded, and passed through a Transformer, with token count $T = (H/p) \times (W/p)$ — e.g. a 336×336 input with 14×14 patches yields $24 \times 24 = 576$ visual tokens. SigLIP replaces the softmax contrastive loss with a sigmoid loss, removing the dependence on in-batch negatives and temperature, so training uses less memory and smaller batches.

the first stage of virtually every modular VLM (LLaVA/Qwen-VL/InternVL); interviews probe “why freeze the vision tower” and the patch-size vs resolution trade-off; high-resolution needs pair it with AnyRes or dynamic-token schemes.

reusing a pretrained CLIP vision tower gives zero-shot alignment for free, and freezing it in early training stages (training only the projector) prevents visual gradients from corrupting text embeddings while saving roughly half the vision tower's memory and compute; the downside is that fixed patches lose fine details — at 336×336 with ViT-14 you only get 576 tokens, so small text/objects are under-encoded, which is why LLaVA-1.6 uses AnyRes grid patching (576 tokens per grid) and Qwen2-VL uses dynamic resolution: trading more tokens for high-res detail.

Test Your Knowledge: Practice Questions for "Vision Encoder"

Single choice pitfall questions with instant feedback and mistake tracking.