M5-009M5: NLP & Large Language ModelsPretraining Objectives & Data CurationMedium
⚡Flashcard Drill➔Mastery:
Data Deduplication Methods and Why It Matters in LLM Pre-training
🎯 Active Recall Self-Check (Draft First, Verify After)
Login-free practice tracker: recall core answer, audit pitfalls, and log review status.
Step 1 · Draft your concise answer & assumptions
Step 2 · Check Pitfalls & Follow-up Inquiries
📐 Mathematical Definition
⚡ One-Sentence Core Answer
Data deduplication removes verbatim and near-duplicate text via exact hash matching and MinHash LSH, preventing model memorization, privacy leakage, and inefficient compute waste.
📌 Key Takeaways
- •Why it matters: duplicate documents cause models to memorize text verbatim, amplify web crawl bias, increase vulnerability to privacy extraction attacks, and waste expensive GPU training cycles
- •Exact deduplication: line-level or document-level SHA-256 hash matching; fast and removes exact duplicates (e.g., license headers, terms of service)
- •Fuzzy (near-duplicate) deduplication: MinHash combined with Locality Sensitive Hashing (LSH) and connected component clustering to remove re-written or templated documents
📐 Mathematical Derivations
Mathematical Mechanism: 1. Jaccard Similarity of Shingle Sets: For document , extract set of -shingles (contiguous character or word -grams) . The similarity between and is: 2. MinHash Theorem (Broder): Let be a random permutation hash function. The probability that the minimum hash value of two sets matches equals their Jaccard similarity: By applying independent hash functions , each document is represented by a compact signature vector . 3. Locality Sensitive Hashing (LSH): Partition the hash values into bands of rows (). Two documents are hashed into the same bucket if they match identically in all rows of at least one band. The probability of collision is: This creates an S-curve that efficiently identifies candidate pairs with in linear time without comparing all document pairs.
🏭 Production Trade-offs
Deep Dive & Engineering Trade-offs: ① Lee et al. (2022) Deduplication Findings: Deduplicating Common Crawl removed of tokens, but reduced memorization by an order of magnitude and resulted in models that trained faster and reached lower perplexity with identical compute. ② Exact vs Fuzzy Granularity: - Exact Hash (SHA-256): Catches boilerplate headers, footers, and copyright notices. - Fuzzy MinHash: Catches scraped news aggregators, slightly modified blog posts, and spam templates. ③ Cross-Split Leakage Removal: Deduplication between the training corpus and standard evaluation benchmarks (MMLU, GSM8K, HumanEval) is mandatory to prevent benchmark data contamination. ④ Distributed Pipeline Architecture: Deduplicating 10TB+ of text requires distributed Spark or Ray clusters; documents are partitioned by LSH bucket IDs and connected components are solved via distributed GraphX. ⑤ Interview Strategy: Derive the MinHash equality theorem, sketch the LSH band probability formula , and quantify the benefits (reduced memorization, privacy protection, compute savings).
⚠️ Common Interview Pitfalls
- ✕Attempting pairwise Jaccard comparisons across billions of documents ( is computationally impossible without LSH)
- ✕Using only exact hashing, which misses slightly modified spam, news reprints, and templated web pages
- ✕Deduplicating too aggressively (e.g., removing common code function signatures or standard idiomatic language patterns)
🎯 Interviewer Follow-up Inquiries
- ?How do the parameters (bands) and (rows) in LSH control false positives and false negatives for target Jaccard threshold ?
- ?Why does training on duplicate data dramatically increase the risk of privacy extraction attacks?
⚡
Test muscle memory on this topic via 3-Layer Flashcards
Active Recall: Mentally formulate Prompt ➔ Math Derivation ➔ Benchmark Implementation in 3 minutes.
📚