CLIP Two-Tower Architecture

CLIP (Contrastive Language-Image Pretraining) is OpenAI's contrastive multimodal model with a two-tower design: an image encoder (ViT such as ViT-B/32, or ResNet) and a text encoder (Transformer), jointly trained on 400M image-text pairs (WIT) to embed images and texts into a shared contrastive space — pulling matched pairs (positives) closer in cosine similarity while pushing other in-batch samples (negatives) apart, with the InfoNCE loss computed symmetrically in both image→text and text→image directions.

zero-shot image classification (76.2% top-1 on ImageNet with no fine-tuning), image-text retrieval, the text-conditioning encoder of text-to-image models (Stable Diffusion reuses CLIP's text tower), and the visual-tower backbone of many VLMs.

classic classifiers require a fixed, hand-labeled label set and retraining for every new class; CLIP turns categories from a finite set into an open language space — at inference you fill candidate labels into prompt templates (“a photo of a {label}”), encode them as text features, and pick the class with the highest similarity to the image feature, enabling zero-shot transfer to arbitrary new datasets from a single pretraining.

Which encoders do CLIP's two towers use, and how do both towers share one contrastive space?

CLIP vs VLM: CLIP only aligns embeddings, why can't it directly chat?

The full zero-shot classification pipeline of CLIP? Role of the prompt template (“a photo of a {label}”)?

Why does CLIP use a symmetric loss (image→text + text→image)? What would one-directional training lose?

CLIP's role in Stable Diffusion: which tower is used, frozen or trainable?