M6-032M6: Multimodal & Generative ModelsVLM Training & EvaluationEasy
⚡Flashcard Drill➔Mastery:

The Canonical Three-Stage Training Pipeline of Multimodal Large Language Models

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M6-033 →
📐 Mathematical Definition
stage1: train connector;stage2: unfreeze LLM;stage3: preference opt.\text{stage1}:\ \text{train connector};\quad \text{stage2}:\ \text{unfreeze LLM};\quad \text{stage3}:\ \text{preference opt.}
⚡ One-Sentence Core Answer
Modern VLMs are trained through a structured three-stage pipeline: pre-training connector alignment on paired image-text captions, visual instruction fine-tuning, and preference alignment via multimodal DPO.

📌 Key Takeaways

  • •
    Stage 1 (Pre-training Alignment): trains lightweight connector weights while freezing vision encoder and LLM on millions of diverse image-caption pairs
  • •
    Stage 2 (Visual Instruction Tuning): unfreezes the LLM (and optionally the vision tower) on high-quality multimodal instruction datasets spanning VQA, OCR, reasoning, and multi-turn chat
  • •
    Stage 3 (Preference Alignment / RLHF): applies Direct Preference Optimization (DPO) on paired chosen/rejected multimodal completions to suppress hallucinations and refine helpfulness

📐 Mathematical Derivations

Mathematical Mechanism: 1. Stage 1: Feature Alignment Pre-training: (a) Data: 500K to 10M image-caption pairs (CC3M, LAION filtered). (b) Status: Vision Tower θv\theta_v Frozen, LLM θllm\theta_{\text{llm}} Frozen, Connector ϕ\phi Trainable. (c) Objective: min⁡ϕ  E(I,X)[−∑t=1∣X∣log⁡P(xt∣x<t,gϕ(fv(I)))]\min_\phi \; \mathbb{E}_{(I, X)} \left[ - \sum_{t=1}^{|X|} \log P(x_t \mid x_{<t}, g_\phi(f_v(I))) \right] Aligns the connector's projection so visual tokens map into valid semantic neighborhoods of the LLM word embedding space. 2. Stage 2: Multimodal Instruction Fine-Tuning: (a) Data: 500K to 2M multi-task instruction pairs (LLaVA-Instruct, ShareGPT4V, DocVQA, MathVista). (b) Status: Connector ϕ\phi Trainable, LLM θllm\theta_{\text{llm}} Trainable, Vision Tower θv\theta_v Frozen / Low LR. (c) Objective: Standard autoregressive loss conditioned on conversation context and visual tokens. 3. Stage 3: Multimodal Direct Preference Optimization (MDPO): (a) Data: Pairs (I,x,yw,yl)(I, x, y_w, y_l) where ywy_w is accurate and yly_l contains visual hallucinations. (b) Objective: LMDPO(θ)=−E(I,x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣I,x)πref(yw∣I,x)−βlog⁡πθ(yl∣I,x)πref(yl∣I,x))]\mathcal{L}_{\text{MDPO}}(\theta) = - \mathbb{E}_{(I, x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid I, x)}{\pi_{\text{ref}}(y_w \mid I, x)} - \beta \log \frac{\pi_\theta(y_l \mid I, x)}{\pi_{\text{ref}}(y_l \mid I, x)} \right) \right] Penalizes responses that assert the existence of non-existent visual entities.

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Rationale for the Three-Stage Staging: Skipping Stage 1 and launching straight into Stage 2 with an uninitialized connector causes severe gradient divergence in the LLM. Skipping Stage 3 leaves the model vulnerable to sycophancy, object existence hallucinations, and repetitive generation loops. ② Data Mixture Ratios in Stage 2: Stage 2 requires a carefully calibrated data recipe: 30% conversational QA, 25% detailed captioning, 20% document/chart OCR, 15% complex multi-step visual math reasoning, and 10% pure text instruction data. Pure text data prevents degradation of coding and math reasoning. ③ Synthetic Data Generation for Stage 3: Hand-labeling preference pairs for multimodal DPO is expensive. Automated pipelines generate negative pairs (yly_l) by prompting frontier models to deliberately introduce factual errors (e.g., swapping colors, adding non-existent objects, modifying chart numerical values) or by selecting rejected completions from early model checkpoints. ④ Compute Budget Allocation: Stage 1 is computationally cheap (only training a 20M parameter MLP for 1 epoch on 600K samples takes <4< 4 hours on 8x A100s). Stage 2 consumes 85-90% of total compute; Stage 3 consumes <10%< 10\% of compute but yields critical gains in human evaluation benchmarks. ⑤ Interview Strategy: Detail the status of components (frozen vs trainable) across all three stages, formulate the loss objectives including MDPO, explain why Stage 1 protects the LLM from random noise, and emphasize the necessity of mixing text-only data into Stage 2.
⚠️ Common Interview Pitfalls
  • ✕
    Omitting text-only instruction datasets during Stage 2 multimodal fine-tuning, triggering severe regression in text reasoning capabilities
  • ✕
    Attempting joint training of the LLM and an uninitialized connector from scratch in a single stage, destabilizing model convergence
  • ✕
    Skipping Stage 3 preference alignment, allowing the model to suffer from high rates of visual hallucination and verbose rambling
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Multimodal Direct Preference Optimization (MDPO) specifically penalize object hallucination in vision-language models?
  • ?
    Why is the compute cost of Stage 1 connector alignment negligible compared to Stage 2 instruction fine-tuning?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM6-031: Impact of Dynamic High Resolution on Optical Character Recognition and Document Understanding📋 Back to BankNext →M6-033: Multimodal Benchmark Taxonomy and Dimensional Capability Evaluation in VLMs