Masked Cross-Entropy Loss

Masked Cross-Entropy (Masked CE) is the standard VLM training loss — sequence-level next-token prediction that computes loss only on assistant (model-answer) text tokens, masking gradients of user-instruction tokens and all vision patch tokens: $$\mathcal{L} = -\sum_{t \in \text{assistant}} \log p_\theta(y_t \mid x_{<t})$$ The input is $x = [v_1, \dots, v_N, u_1, \dots, u_m, a_1, \dots, a_L]$ (vision patches → user instruction → assistant answer); causal attention sees all tokens, but only $t \in \text{assistant}$ positions contribute loss — exactly the LLM SFT labels mechanism where non-supervised positions are set to $\text{ignore\_index}$ (e.g. -100). Why mask: (1) vision patch tokens have no "next token" supervision target (they come from the frozen encoder, they are not generation targets) — CE on them forces patch-sequence prediction, pointless and harmful to text learning; (2) user instructions are not content the model should generate — predicting them teaches the model to parrot the user; (3) compute efficiency — with high-resolution inputs, vision tokens dominate the sequence (336×336, patch 14 → 576 vision tokens, often 85–90% of the sequence), so masking cuts backward-computation roughly an order of magnitude. An optional $1/L$ normalizes by supervised-token count.

the standard objective across all VLM stages (alignment/pretraining/SFT); interview follow-ups on how the multimodal loss is written and whether vision tokens are supervised; it also links to why hallucination correlates with missing vision gradients (see the multimodal-hallucination card).

naively computing CE over the whole sequence makes the model learn to predict image patches and echo user questions, wasting 85–90% of compute and degrading generation quality; Masked CE concentrates supervision on what the model should actually generate (assistant text), mirroring the ignore_index mechanism of LLM chat templates — the design that unifies VLM and LLM training objectives.

Write the masked-CE formula; which tokens are masked and why?

What goes wrong if you compute CE on vision patch tokens?

With vision tokens at 85–90% of the sequence, how does masking affect compute (336×336→576 patches)?

How does masked CE correspond to the ignore_index=-100 mechanism of LLM SFT?

Why mask user instructions too? What would the model learn otherwise?