VLM Projector MLP/Q-Former

The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — $h = \text{MLP}(z_{vis})$ maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576→576); ② BLIP-2's Q-Former — a set of learnable queries $\mathbf{q} \in \mathbb{R}^{32 \times d}$ extract information from visual features via cross-attention, compressing 576 visual tokens into 32 query tokens; ③ C-Abstractor — downsamples with $2 \times 2$ strided convolutions, dividing token count by 4.

the second stage of every modular VLM; interviews ask “why does LLaVA use an MLP while BLIP-2 uses Q-Former” and how to pick the compression ratio.

visual features (1024-dim CLIP space) and text embeddings (4096-dim LLM space) live in different spaces and cannot be concatenated directly; the MLP route compresses nothing and preserves spatial correspondence, suiting detail-hungry tasks (OCR/grounding), but 576 tokens inflate attention cost in the LLM; Q-Former's cross-attention queries compress 18× (576→32) for faster inference at the cost of spatial detail — which is why later Qwen-VL models went back to “MLP + a bigger vision tower”, showing the projector is the fidelity-vs-compression trade-off point.

Write the MLP projection formula $h = \text{MLP}(z_{vis})$; how do token count and dims change in LLaVA?

How does Q-Former compress 576 visual tokens into 32? Role of learnable queries?

MLP projection vs Q-Former compression: trade-offs? Which tasks favor which?

How is the projector trained in the alignment stage? Is it updated together with the LLM?

How does C-Abstractor downsample? How many tokens does it save?