M5-073M5: NLP & Large Language ModelsRAG End-to-End ArchitectureEasy
⚡Flashcard Drill➔Mastery:

Architecture and Components of a Complete RAG System

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M5-074 →
📐 Mathematical Definition
RAG: chunk→embed→index→retrieve→rerank→generate\text{RAG}:\ \text{chunk}\to\text{embed}\to\text{index}\to\text{retrieve}\to\text{rerank}\to\text{generate}
⚡ One-Sentence Core Answer
A production RAG system orchestrates document ingestion, chunking, dense-sparse hybrid retrieval, cross-encoder reranking, and context-augmented generation to provide accurate, grounded responses from external knowledge bases.

📌 Key Takeaways

  • •
    Five core stages: Ingestion & Chunking →\to Indexing (Dense + Sparse) →\to Hybrid Retrieval →\to Cross-Encoder Reranking →\to Generation with Context
  • •
    Decoupling storage: splits document storage (relational/blob store for full text) from search indices (vector database for embeddings, inverted index for BM25)
  • •
    Precision vs Recall funnel: retrieval casts a wide net (K=50–100K=50\text{--}100 candidate chunks); reranker filters to top-kk (k=3–5k=3\text{--}5) high-relevance chunks to prevent context dilution

📐 Mathematical Derivations

Mathematical Mechanism: The RAG System Pipeline Funnel: Given raw document corpus D={D1,…,DM}\mathcal{D} = \{D_1, \dots, D_M\} and user query qq: 1. Ingestion & Parsing: Parse PDF/HTML/Markdown, clean boilerplate, extract tables. 2. Chunking: Split documents into chunks C={c1,…,cN}C = \{c_1, \dots, c_N\} via recursive semantic splitting. 3. Dual Indexing: - Dense Vector Index: vi=E(ci)∈Rdv_i = E(c_i) \in \mathbb{R}^d indexed in HNSW / Milvus. - Sparse Lexical Index: Term inverted index for BM25 in Elasticsearch. 4. Hybrid Retrieval (Top-$K$): Retrieve top-KK candidates (K=50K=50) combining dense cosine similarity and sparse BM25 scores via Reciprocal Rank Fusion (RRF). 5. Cross-Encoder Reranking (Top-$k$): Score each candidate jointly with query: si=CrossEncoder(q,ci)s_i = \text{CrossEncoder}(q, c_i), selecting top-kk (k=5k=5) chunks. 6. Prompt Synthesis & Generation: y∼PLLM(⋅∣Prompt(q,{c(1),…,c(k)}))y \sim P_{\text{LLM}}(\cdot \mid \text{Prompt}(q, \{c_{(1)}, \dots, c_{(k)}\}))

🏭 Production Trade-offs

Deep Dive & Engineering Trade-offs: ① The Multi-Stage Funnel Rationale: Embedding retrieval is fast (O(1)O(1) via ANN) but coarse; cross-encoder reranking is accurate (O(L2)O(L^2) cross-attention) but slow. Combining fast candidate generation (K=50K=50) with a heavy reranker on 50 chunks achieves <50 ms<50\text{ ms} latency with high precision. ② Small Chunks for Retrieval, Large Chunks for Generation (Parent Document Retrieval): Retrieve small 128-token chunks (dense semantic precision), but pass their parent 1024-token section into the LLM context, giving the model full surrounding context without retrieval noise. ③ Metadata Filtering (Pre-filtering vs Post-filtering): Attach structured metadata (user_id, department, date) to chunks. Pre-filtering restricts vector search to the authorized tenant metadata subspace, enforcing enterprise security and speeding up search. ④ Context Window Budgeting: Shoving 20 chunks into the context window triggers the 'Lost in the Middle' phenomenon. A high-quality RAG system uses fewer, higher-relevance chunks (k=3–5k=3\text{--}5). ⑤ Interview Strategy: Draw the 5-stage funnel diagram, contrast Bi-Encoder retrieval vs Cross-Encoder reranking, and explain Parent Document Retrieval.
⚠️ Common Interview Pitfalls
  • ✕
    Sending raw retrieved chunks directly to the LLM without cross-encoder reranking (results in noisy context and hallucination)
  • ✕
    Using naive fixed-character chunking that splits sentences or code blocks mid-word
  • ✕
    Over-stuffing the LLM context window with 30+ chunks, diluting the attention mechanism
🎯 Interviewer Follow-up Inquiries
  • ?
    How does Parent Document Retrieval combine fine-grained vector matching with broad context comprehension?
  • ?
    What is the operational latency breakdown across the 5 stages of a production RAG pipeline?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM5-072: Prompt Formatting Sensitivity and the Impact of Chat Templates📋 Back to BankNext →M5-074: Document Chunking Strategies and Trade-offs in RAG