M6-032M6: Multimodal & Generative ModelsVLM Training & EvaluationEasy
⚡Flashcard Drill➔Mastery:
The Canonical Three-Stage Training Pipeline of Multimodal Large Language Models
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Modern VLMs are trained through a structured three-stage pipeline: pre-training connector alignment on paired image-text captions, visual instruction fine-tuning, and preference alignment via multimodal DPO.
📌 Key Takeaways
- •Stage 1 (Pre-training Alignment): trains lightweight connector weights while freezing vision encoder and LLM on millions of diverse image-caption pairs
- •Stage 2 (Visual Instruction Tuning): unfreezes the LLM (and optionally the vision tower) on high-quality multimodal instruction datasets spanning VQA, OCR, reasoning, and multi-turn chat
- •Stage 3 (Preference Alignment / RLHF): applies Direct Preference Optimization (DPO) on paired chosen/rejected multimodal completions to suppress hallucinations and refine helpfulness
📐 Mathematical Derivations
Mathematical Mechanism: 1. Stage 1: Feature Alignment Pre-training: (a) Data: 500K to 10M image-caption pairs (CC3M, LAION filtered). (b) Status: Vision Tower Frozen, LLM Frozen, Connector Trainable. (c) Objective: Aligns the connector's projection so visual tokens map into valid semantic neighborhoods of the LLM word embedding space. 2. Stage 2: Multimodal Instruction Fine-Tuning: (a) Data: 500K to 2M multi-task instruction pairs (LLaVA-Instruct, ShareGPT4V, DocVQA, MathVista). (b) Status: Connector Trainable, LLM Trainable, Vision Tower Frozen / Low LR. (c) Objective: Standard autoregressive loss conditioned on conversation context and visual tokens. 3. Stage 3: Multimodal Direct Preference Optimization (MDPO): (a) Data: Pairs where is accurate and contains visual hallucinations. (b) Objective: Penalizes responses that assert the existence of non-existent visual entities.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① The Rationale for the Three-Stage Staging: Skipping Stage 1 and launching straight into Stage 2 with an uninitialized connector causes severe gradient divergence in the LLM. Skipping Stage 3 leaves the model vulnerable to sycophancy, object existence hallucinations, and repetitive generation loops. ② Data Mixture Ratios in Stage 2: Stage 2 requires a carefully calibrated data recipe: 30% conversational QA, 25% detailed captioning, 20% document/chart OCR, 15% complex multi-step visual math reasoning, and 10% pure text instruction data. Pure text data prevents degradation of coding and math reasoning. ③ Synthetic Data Generation for Stage 3: Hand-labeling preference pairs for multimodal DPO is expensive. Automated pipelines generate negative pairs () by prompting frontier models to deliberately introduce factual errors (e.g., swapping colors, adding non-existent objects, modifying chart numerical values) or by selecting rejected completions from early model checkpoints. ④ Compute Budget Allocation: Stage 1 is computationally cheap (only training a 20M parameter MLP for 1 epoch on 600K samples takes hours on 8x A100s). Stage 2 consumes 85-90% of total compute; Stage 3 consumes of compute but yields critical gains in human evaluation benchmarks. ⑤ Interview Strategy: Detail the status of components (frozen vs trainable) across all three stages, formulate the loss objectives including MDPO, explain why Stage 1 protects the LLM from random noise, and emphasize the necessity of mixing text-only data into Stage 2.
⚠️ Common Interview Pitfalls
- ✕Omitting text-only instruction datasets during Stage 2 multimodal fine-tuning, triggering severe regression in text reasoning capabilities
- ✕Attempting joint training of the LLM and an uninitialized connector from scratch in a single stage, destabilizing model convergence
- ✕Skipping Stage 3 preference alignment, allowing the model to suffer from high rates of visual hallucination and verbose rambling
🎯 Interviewer Follow-up Inquiries
- ?How does Multimodal Direct Preference Optimization (MDPO) specifically penalize object hallucination in vision-language models?
- ?Why is the compute cost of Stage 1 connector alignment negligible compared to Stage 2 instruction fine-tuning?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚