Native Any-to-Any Multimodal

Native (homogeneous) multimodal models unify images, audio and text into one discrete token space and run autoregressive next-token prediction on a single Transformer backbone: Chameleon (Meta) quantizes images into discrete VQGAN image tokens mixed with text tokens on a shared backbone; GPT-4o tokenizes audio as well (audio in → text/audio tokens out) with all modalities sharing attention, reaching an end-to-end latency of about 320ms for real-time voice conversations; every modality shares one objective $\mathcal{L} = -\log p(x_{t+1} \mid x_{1:t})$.

real-time voice assistants (GPT-4o voice), any-to-any generation, agents needing joint reasoning across hearing/seeing/reading; interviews ask about the native vs modular (LLaVA-style) trade-off.

modular VLMs are heterogeneous assemblies (encoder + projector + LLM) trained stage-wise with different losses, exchanging information only at the projector, which limits joint reasoning and low-latency streaming; native models train everything end-to-end in a shared backbone with a single objective, so modalities interact freely mid-generation (images can steer ongoing text) and streaming is natural — at the cost of trickier data mixing and training stability, with text ability potentially diluted by multimodal data (needing extra protective strategies).

Native multimodal (GPT-4o/Chameleon) vs modular VLM (LLaVA): core differences?

How are images discretized into tokens for autoregressive training? Role of VQGAN?

Why is GPT-4o voice low-latency? Benefits of audio tokenization?

Cost of a unified token space? Why may text ability get diluted?

How do native models handle video/long context? Token-count and compute challenges?