👁️ MultimodalID: vlm-projector

VLM Projector MLP/Q-Former

Q: Write the MLP projection formula $h = \text{MLP}(z_{vis})$; how do token count and dims change in LLaVA?

【VLM Projector MLP/Q-Former 核心考点 1】Write the MLP projection formula $h = \text{MLP}(z_{vis})$; how do token count and dims change in LLaVA?。技术解析：Definition: The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — h = \text{MLP}(z_{vis}) maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/vlm-projector

Q: How does Q-Former compress 576 visual tokens into 32? Role of learnable queries?

【VLM Projector MLP/Q-Former 核心考点 2】How does Q-Former compress 576 visual tokens into 32? Role of learnable queries?。技术解析：Definition: The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — h = \text{MLP}(z_{vis}) maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/vlm-projector

Q: MLP projection vs Q-Former compression: trade-offs? Which tasks favor which?

【VLM Projector MLP/Q-Former 核心考点 3】MLP projection vs Q-Former compression: trade-offs? Which tasks favor which?。技术解析：Definition: The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — h = \text{MLP}(z_{vis}) maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/vlm-projector

Q: How is the projector trained in the alignment stage? Is it updated together with the LLM?

【VLM Projector MLP/Q-Former 核心考点 4】How is the projector trained in the alignment stage? Is it updated together with the LLM?。技术解析：Definition: The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — h = \text{MLP}(z_{vis}) maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/vlm-projector

Q: How does C-Abstractor downsample? How many tokens does it save?

【VLM Projector MLP/Q-Former 核心考点 5】How does C-Abstractor downsample? How many tokens does it save?。技术解析：Definition: The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP — h = \text{MLP}(z_{vis}) maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/Multimodal/card/vlm-projector

VLM 投影器 MLP/Q-Former

🎯Core Definition

The projector maps visual features into the LLM's text embedding space; three mainstream routes: ① LLaVA's linear/two-layer MLP —

h = \text{MLP}(z_{vis})

maps each visual token independently (typically 1024→4096 dims), preserving token count and spatial structure (576→576); ② BLIP-2's Q-Former — a set of learnable queries

\mathbf{q} \in \mathbb{R}^{32 \times d}

extract information from visual features via cross-attention, compressing 576 visual tokens into 32 query tokens; ③ C-Abstractor — downsamples with

2 \times 2

strided convolutions, dividing token count by 4.

💡Use Cases

the second stage of every modular VLM; interviews ask “why does LLaVA use an MLP while BLIP-2 uses Q-Former” and how to pick the compression ratio.

⚡Key Problems Solved

visual features (1024-dim CLIP space) and text embeddings (4096-dim LLM space) live in different spaces and cannot be concatenated directly; the MLP route compresses nothing and preserves spatial correspondence, suiting detail-hungry tasks (OCR/grounding), but 576 tokens inflate attention cost in the LLM; Q-Former's cross-attention queries compress 18× (576→32) for faster inference at the cost of spatial detail — which is why later Qwen-VL models went back to “MLP + a bigger vision tower”, showing the projector is the fidelity-vs-compression trade-off point.

🎯5 High-Frequency Exam Points

Write the MLP projection formula