Pre-Training Data Engineering

Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are candidate near-duplicate pairs. Quality filtering scores documents by PPL (perplexity) of a pretrained LM; the domain mix blends Web:Code:Math at a set ratio; the annealing phase (last ~5% of tokens) injects high-quality and synthetic data such as Evol-Instruct's iteratively evolved instructions.

data pipelines before pretraining, cleaning RLHF preference data, model reproduction and data-ablation analysis.

trillion-scale corpora without dedup cause repeated memorization, wasted compute and benchmark contamination. MinHash+LSH cuts near-duplicate search from $O(n^2)$ to near-linear, scaling to trillions of tokens; PPL filtering removes ~30%-50% of low-quality docs; annealing raises the high-quality share above 50%, gaining several points on downstream benchmarks at the same ~1.4T-token compute budget.

Test Your Knowledge: Practice Questions for "Pre-Training Data Engineering"

Single choice pitfall questions with instant feedback and mistake tracking.