ColPali Visual Retrieval

ColPali feeds PDF pages directly into a VLM (PaliGemma) to obtain patch-level vectors, encodes the query into token-level vectors with an LLM, and scores with ColBERT-style Late-Interaction MaxSim: $$\text{sim}(q, d) = \sum_i \max_j \langle q_i, d_j \rangle$$ where $q_i$ is the $i$-th query token vector and $d_j$ is the $j$-th image patch vector of the document — each query token takes the max cosine similarity over all patches, then sums. Query and document vectors are computed independently and only interact at retrieval time (Late-Interaction), so document vectors are precomputed offline.

visual document/PDF QA and retrieval (e.g. paper search); a common interview question is 'why does it skip OCR?'.

the traditional OCR-then-retrieve pipeline loses layout, chart, and formula information and suffers cascading errors; ColPali retrieves directly over page images, skipping OCR and layout parsing — on document-retrieval benchmarks it lifts F1 by 7–40% (up to +20 points) over 'OCR + text retrieval' pipelines, with faster indexing and smaller storage.

Write ColPali/ColBERT's Late-Interaction MaxSim scoring formula and explain each term?

ColPali vs the 'OCR + text retrieval' pipeline: differences and gains (with metrics)?

Why does ColPali use VLM patch-level vectors instead of CLIP's global image vector?

Late-interaction vs dual-tower global-vector retrieval: precision/latency tradeoff?

How are ColPali document vectors precomputed offline? Online retrieval and rerank pipeline?