VLM Architecture

The standard VLM (Vision-Language Model) recipe is three parts — vision encoder + projector + LLM: the vision encoder (CLIP ViT) patchifies the image into visual features $z_{vis}$, the projector (MLP/Q-Former) maps them into the LLM's text embedding space, and the LLM generates text autoregressively, trained by cross-entropy on assistant tokens: $\mathcal{L} = -\sum_t \log p(y_t \mid y_{<t}, \mathbf{v})$. Fusion placement defines the family: LLaVA/Qwen-VL fuse at the LLM input (projected visual tokens prepended to the input sequence); BLIP-2 compresses via Q-Former cross-attention before injection; Flamingo inserts cross-attention layers inside the LLM (interleaved fusion).

visual question answering (VQA), image captioning, OCR/document understanding, chart reasoning, video understanding, and as the perception layer of multimodal agents; interviews ask about the differences between the three fusion placements and the rationale for each.

training a truly native multimodal transformer from scratch is prohibitively expensive in data and compute; the modular recipe reuses an LLM's language ability and CLIP's vision ability, requiring training only a lightweight projector (millions of parameters) to connect vision into language — three-stage training (alignment → interleaved pretraining → visual SFT) reduces the cost from training an LLM to fine-tuning one, while inheriting open-source strength in both modalities.

The VLM three-part recipe: responsibilities of encoder / projector / LLM?

LLaVA (input-level fusion) vs Flamingo (layer-level cross-attention): differences?

Why project visual features into the text embedding space? How is the dimension mismatch handled?

Why is the VLM loss cross-entropy over text tokens only? Is there no loss on images?

Native multimodal (GPT-4o/Chameleon) vs modular VLM (LLaVA): architecture trade-offs?