M4-046M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:
Fusing Attention with External Memory and Retrieval: kNN-LM and RETRO
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
kNN-LM interpolates softmax output with nearest-neighbor vector retrieval; RETRO integrates retrieved text chunks directly into cross-attention blocks.
📌 Key Takeaways
- •kNN-LM (Khandelwal et al., ICLR 2020): queries an offline datastore of billions of cached hidden states via FAISS at test time
- •Dual probability interpolation: ; zero training required
- •RETRO (Borgeaud et al., DeepMind 2022): pretrains with retrieved document chunks integrated via Chunked Cross-Attention
📐 Mathematical Derivations
Mathematical Formulations:
① kNN-LM (Non-parametric Nearest Neighbors):
1. Datastore Construction: Pass training corpus through a pretrained LM. Store key-value pairs: , where is the final hidden state context vector and is the target next token (billions of vectors indexed in FAISS).
2. Inference Query: For current context , compute query . Retrieve top- nearest neighbors .
3. Compute non-parametric probability distribution using softmax over negative L2 distances:
.
4. Blend probabilities: .
② RETRO (Retrieval-Enhanced Transformer):
Splits input into chunks of 64 tokens. Retrieves similar 64-token chunks from an external 3-trillion-token database. Interleaves Chunked Cross-Attention into Transformer blocks, allowing a 7B model to match the factual accuracy of a 175B model.
① kNN-LM (Non-parametric Nearest Neighbors):
1. Datastore Construction: Pass training corpus through a pretrained LM. Store key-value pairs: , where is the final hidden state context vector and is the target next token (billions of vectors indexed in FAISS).
2. Inference Query: For current context , compute query . Retrieve top- nearest neighbors .
3. Compute non-parametric probability distribution using softmax over negative L2 distances:
.
4. Blend probabilities: .
② RETRO (Retrieval-Enhanced Transformer):
Splits input into chunks of 64 tokens. Retrieves similar 64-token chunks from an external 3-trillion-token database. Interleaves Chunked Cross-Attention into Transformer blocks, allowing a 7B model to match the factual accuracy of a 175B model.
🏭 Production Trade-offs
Inference Latency Trade-off: kNN-LM dramatically boosts factual accuracy on rare entities without fine-tuning, but querying billion-scale FAISS indexes adds 10–50ms latency per generated token. Modern production systems prefer Retrieval-Augmented Generation (RAG) in prompts.
⚠️ Common Interview Pitfalls
- ✕Deploying kNN-LM without vector quantization (IVF-PQ), causing multi-terabyte memory exhaustion on inference servers
- ✕Assuming kNN-LM modifies model weights; kNN-LM is a strictly non-parametric post-processing ensemble
🎯 Interviewer Follow-up Inquiries
- ?How does kNN-LM achieve massive perplexity improvements on rare entities without updating a single model weight?
- ?How does RETRO's Chunked Cross-Attention avoid autoregressive causal leakage during generation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚