Text Embeddings

Text embeddings map tokens or whole sentences into dense, low-dimensional real vectors so semantically similar texts lie close in vector space. Token-level embeddings come from the vocabulary embedding table $E \in \mathbb{R}^{V \times d}$ (V = vocab size, d = hidden dim); sentence-level embeddings typically take the CLS token output or mean-pool all token vectors, with cosine similarity measuring semantic closeness.

RAG retrieval (encode the document corpus offline into a vector store, nearest-neighbor search for queries), semantic search and QA recall, clustering/dedup, fine-tuning embedding models for domain retrieval — the underlying representation of vector DBs (ANN search) feeding KB question answering.

bag-of-words / one-hot vectors are V-dimensional (100K-level) and semantics-free — they cannot capture relations like king − man + woman ≈ queen; embeddings compress semantics into $d \ll V$ dense space with cosine similarity in $[-1, 1]$, replacing keyword matching with semantic search. With V=100K, d=4096 the embedding table alone is ~400M params (~1.6GB FP32), showing why high-dimensional sparse representations are infeasible.

Test Your Knowledge: Practice Questions for "Text Embeddings"

Single choice pitfall questions with instant feedback and mistake tracking.