Interleaved Training Data

Interleaved training data is the data pyramid for VLM pretraining, organized as image-text pairs → interleaved documents → video; the mixing ratio effectively sets the capability ceiling. Tier 1 — image-text pairs (CC3M/LAION-5B): one caption per image, strongest alignment signal but limited stylistic diversity; teaches "say what you see". Tier 2 — interleaved documents (OBELICS ~115M web docs / MMC4): multiple images woven into text, teaching multi-image comparison, cross-modal referencing and long-context reasoning. Tier 3 — video-text (WebVid-10M): frame sampling (typically 4–8 frames/clip) extends to the temporal axis. Vision token budget: with ViT patch size $p$, one image yields $N_{\text{vision}} = \lceil H/p \rceil \times \lceil W/p \rceil$ tokens (LLaVA-1.5: 336×336, patch 14 → 24×24 = 576); video multiplies by frames $T$: $N_{\text{vision}} = T \times \lceil H/p \rceil \times \lceil W/p \rceil$.

data organization and ratio design for VLM pretraining (LLaVA/Qwen-VL/InternVL); interview follow-ups on where data comes from, how it is mixed, and why; also explains why a model excels at multi-image or temporal reasoning.

pure pair data gives narrow capability (strong alignment, no multi-image reasoning), pure interleaved docs give weak alignment; the three-tier pyramid mixed in proportion (e.g. Flamingo's pairs:interleaved:video) yields alignment + multi-image/long-context reasoning + temporal understanding simultaneously — too many pairs leaves a single-image QA model, too many interleaved docs hurts alignment, so the data ratio is the capability ceiling.

What is the three-tier data pyramid for VLM pretraining and the role of each tier?

How does the mixing ratio of pairs/interleaved/video set the VLM capability ceiling?

What are OBELICS/MMC4/WebVid-10M? What unique ability does interleaved-document pretraining bring?

How many vision tokens does a 336×336 image produce at patch=14? How do video frames scale the budget?

Interleaved vs paired data: which matters more for multi-image reasoning and long context, and why?