Text Embeddings

Text embeddings map tokens or whole sentences into dense, low-dimensional real vectors so semantically similar texts lie close in vector space. Token-level embeddings come from the vocabulary embedding table $E \in \mathbb{R}^{V \times d}$ (V = vocab size, d = hidden dim); sentence-level embeddings typically take the CLS token output or mean-pool all token vectors, with cosine similarity measuring semantic closeness.

RAG retrieval (encode the document corpus offline into a vector store, nearest-neighbor search for queries), semantic search and QA recall, clustering/dedup, fine-tuning embedding models for domain retrieval — the underlying representation of vector DBs (ANN search) feeding KB question answering.

bag-of-words / one-hot vectors are V-dimensional (100K-level) and semantics-free — they cannot capture relations like king − man + woman ≈ queen; embeddings compress semantics into $d \ll V$ dense space with cosine similarity in $[-1, 1]$, replacing keyword matching with semantic search. With V=100K, d=4096 the embedding table alone is ~400M params (~1.6GB FP32), showing why high-dimensional sparse representations are infeasible.

Token-level vs sentence-level embeddings? When CLS vs mean pooling?

Why cosine similarity over dot product or L2 distance? Its range?

Embedding's role in RAG: offline indexing, online ANN search, reranking?

Estimating params/memory of the vocab embedding table?

Static (Word2Vec) vs contextual representations (Transformer outputs)?