ColBERT Late Interaction

ColBERT (Contextualized Late Interaction over BERT) is a multi-vector retrieval and re-ranking architecture that fuses the speed of Bi-Encoders with the precision of Cross-Encoders; instead of compressing documents into a single dense vector, ColBERT maintains independent contextualized token embeddings for all tokens; during scoring, it bypasses heavy cross-attention layers, computing relevance via a lightweight MaxSim operator $S(Q, D) = \sum_{i \in |Q|} \max_{j \in |D|} (E(q_i) \cdot E(d_j))$ that sums maximum cosine alignments for each query token across document tokens.

Ultra-low-latency (sub-10ms) precision re-ranking, multi-vector end-to-end search (RAGatouille), and visual multi-modal page indexing (ColPali).

Heavy Cross-Encoders ($O(N \cdot L^2)$) induce 100ms+ latency, blocking high-concurrency scaling; single-vector Bi-Encoders lose fine-grained token alignments; ColBERT's MaxSim reduces scoring to batched matrix dot products, delivering Cross-Encoder quality at millisecond speeds.

Derive the ColBERT MaxSim operator and explain how it captures both soft lexical matches and deep contextual semantics?

How does ColBERTv2 apply residual compression and centroid quantization to reduce multi-vector storage by 6x-10x?

Compare ColBERT (Late Interaction), Bi-Encoder, and Cross-Encoder across latency, VRAM footprint, and NDCG?

How does the PLAID search engine achieve millisecond-level retrieval via 3-stage pruning (centroid, BoW, exact MaxSim)?

How does ColPali extend ColBERT late interaction to Vision LLMs (PaliGemma) for native multi-vector screenshot retrieval?