M4-046M4: Sequences & TransformersAttention Variants (MHA / MQA / GQA)Hard
⚡Flashcard Drill➔Mastery:

Fusing Attention with External Memory and Retrieval: kNN-LM and RETRO

🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
Next: M4-047 →
📐 Mathematical Definition
kNN-LM: p=λpLM+(1−λ)pkNN;RETRO: chunked cross-attention\text{kNN-LM}:\ p=\lambda p_{\text{LM}}+(1-\lambda)p_{\text{kNN}};\qquad \text{RETRO}:\ \text{chunked cross-attention}
⚡ One-Sentence Core Answer
kNN-LM interpolates softmax output with nearest-neighbor vector retrieval; RETRO integrates retrieved text chunks directly into cross-attention blocks.

📌 Key Takeaways

  • •
    kNN-LM (Khandelwal et al., ICLR 2020): queries an offline datastore of billions of cached hidden states via FAISS at test time
  • •
    Dual probability interpolation: P(yt)=λPkNN(yt)+(1−λ)PLM(yt)P(y_t) = \lambda P_{\text{kNN}}(y_t) + (1 - \lambda) P_{\text{LM}}(y_t); zero training required
  • •
    RETRO (Borgeaud et al., DeepMind 2022): pretrains with retrieved document chunks integrated via Chunked Cross-Attention

📐 Mathematical Derivations

Mathematical Formulations:
① kNN-LM (Non-parametric Nearest Neighbors):
1. Datastore Construction: Pass training corpus through a pretrained LM. Store key-value pairs: D={(f(ci),wi)}\mathcal{D} = \{(f(c_i), w_i)\}, where f(ci)∈Rdf(c_i) \in \mathbb{R}^d is the final hidden state context vector and wiw_i is the target next token (billions of vectors indexed in FAISS).
2. Inference Query: For current context xx, compute query q=f(x)q = f(x). Retrieve top-kk nearest neighbors N\mathcal{N}.
3. Compute non-parametric probability distribution using softmax over negative L2 distances:
PkNN(w∣x)∝∑(hj,wj)∈N,wj=wexp⁡(−∥q−hj∥22T)P_{\text{kNN}}(w \mid x) \propto \sum_{(h_j, w_j) \in \mathcal{N}, w_j = w} \exp\left( -\frac{\|q - h_j\|_2^2}{T} \right).
4. Blend probabilities: Pfinal(w∣x)=λPkNN(w∣x)+(1−λ)PLM(w∣x)P_{\text{final}}(w \mid x) = \lambda P_{\text{kNN}}(w \mid x) + (1 - \lambda) P_{\text{LM}}(w \mid x).
② RETRO (Retrieval-Enhanced Transformer):
Splits input into chunks of 64 tokens. Retrieves similar 64-token chunks from an external 3-trillion-token database. Interleaves Chunked Cross-Attention into Transformer blocks, allowing a 7B model to match the factual accuracy of a 175B model.

🏭 Production Trade-offs

Inference Latency Trade-off: kNN-LM dramatically boosts factual accuracy on rare entities without fine-tuning, but querying billion-scale FAISS indexes adds 10–50ms latency per generated token. Modern production systems prefer Retrieval-Augmented Generation (RAG) in prompts.
⚠️ Common Interview Pitfalls
  • ✕
    Deploying kNN-LM without vector quantization (IVF-PQ), causing multi-terabyte memory exhaustion on inference servers
  • ✕
    Assuming kNN-LM modifies model weights; kNN-LM is a strictly non-parametric post-processing ensemble
🎯 Interviewer Follow-up Inquiries
  • ?
    How does kNN-LM achieve massive perplexity improvements on rare entities without updating a single model weight?
  • ?
    How does RETRO's Chunked Cross-Attention avoid autoregressive causal leakage during generation?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
🎯 Drill This Card Now➔
📚

Associated Knowledge Base Guides & Mindmaps

← PreviousM4-045: Alternatives to Softmax Attention: Sigmoid Attention and Normalization Variants📋 Back to BankNext →M4-047: FlashAttention Core Philosophy: IO-Awareness, SRAM Tiling, and Memory Hierarchy