👁️ MultimodalID:
clip-architectureCLIP Two-Tower Architecture
CLIP 双塔架构
🎯Core Definition
CLIP (Contrastive Language-Image Pretraining) is OpenAI's contrastive multimodal model with a two-tower design: an image encoder (ViT such as ViT-B/32, or ResNet) and a text encoder (Transformer), jointly trained on 400M image-text pairs (WIT) to embed images and texts into a shared contrastive space — pulling matched pairs (positives) closer in cosine similarity while pushing other in-batch samples (negatives) apart, with the InfoNCE loss computed symmetrically in both image→text and text→image directions.
💡Use Cases
zero-shot image classification (76.2% top-1 on ImageNet with no fine-tuning), image-text retrieval, the text-conditioning encoder of text-to-image models (Stable Diffusion reuses CLIP's text tower), and the visual-tower backbone of many VLMs.
⚡Key Problems Solved
classic classifiers require a fixed, hand-labeled label set and retraining for every new class; CLIP turns categories from a finite set into an open language space — at inference you fill candidate labels into prompt templates (“a photo of a {label}”), encode them as text features, and pick the class with the highest similarity to the image feature, enabling zero-shot transfer to arbitrary new datasets from a single pretraining.
🎯5 High-Frequency Exam Points
1
Which encoders do CLIP's two towers use, and how do both towers share one contrastive space?
2
CLIP vs VLM: CLIP only aligns embeddings, why can't it directly chat?
3
The full zero-shot classification pipeline of CLIP? Role of the prompt template (“a photo of a {label}”)?
4
Why does CLIP use a symmetric loss (image→text + text→image)? What would one-directional training lose?
5
CLIP's role in Stable Diffusion: which tower is used, frozen or trainable?
📖 In-depth Guide:📄 clip-and-contrastive-learning →
Updated 2026-08-12🎯
🚀 Start Card Practice ➔Test Your Knowledge: Practice Questions for "CLIP Two-Tower Architecture"
Single choice pitfall questions with instant feedback and mistake tracking.