CLIP Applications

CLIP learns a shared image-text semantic space, making it a zero-shot multimodal foundation with four application families: ① zero-shot retrieval/classification — text→image search, 76.2% zero-shot top-1 on ImageNet; ② text-conditioning encoder for text-to-image — Stable Diffusion feeds prompts through CLIP's text tower to get text embeddings that steer generation (and uses the visual tower for similarity-based training); ③ multimodal embedding backbone — image retrieval/dedup, caption-quality filtering, joint image-text retrieval for multimodal RAG; ④ feedback signal — CLIP score as a proxy for generation quality evaluation and image reward models.

e-commerce image/text search, image-text consistency moderation, SD/ComfyUI workflows, multimodal retrieval-augmented generation; interviews often ask “what else can CLIP embeddings do?”

a single similarity computation replaces the classic pipeline of classification heads + fixed label sets + per-task fine-tuning — retrieval, classification, ranking, conditional generation and quality scoring all reuse one contrastive space, and the model plugs natively into the T2I/VLM/RAG ecosystem (SD consumes the text tower directly, LLaVA reuses the visual tower weights), saving most of the training cost from scratch.

CLIP's role in Stable Diffusion? Why do text-to-image models universally reuse the CLIP text tower?

How to use CLIP embeddings for RAG/retrieval? How is joint image-text retrieval done?

Limitations of CLIP zero-shot classification? What does the gap to linear probe reveal?

CLIP score for generation evaluation: mechanism? How well does it correlate with human judgment?

CLIP retrieval failure modes (fine-grained, abstract semantics, long-tail)? Mitigations?