⚡ LLMsID: pretrain-data-engineering

Pre-Training Data Engineering

Q: How does MinHash+LSH work? Why is approximate dedup sufficient?

【Pre-Training Data Engineering 核心考点 1】How does MinHash+LSH work? Why is approximate dedup sufficient?。技术解析：Definition: Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are ca... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/pretrain-data-engineering

Q: How does PPL filtering work? Is low PPL always high quality?

【Pre-Training Data Engineering 核心考点 2】How does PPL filtering work? Is low PPL always high quality?。技术解析：Definition: Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are ca... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/pretrain-data-engineering

Q: How to design the domain mix (Web:Code:Math)? Which capabilities does it shape?

【Pre-Training Data Engineering 核心考点 3】How to design the domain mix (Web:Code:Math)? Which capabilities does it shape?。技术解析：Definition: Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are ca... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/pretrain-data-engineering

Q: What is the annealing phase? Why inject high-quality/synthetic data?

【Pre-Training Data Engineering 核心考点 4】What is the annealing phase? Why inject high-quality/synthetic data?。技术解析：Definition: Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are ca... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/pretrain-data-engineering

Q: Synthetic data (Evol-Instruct etc.) and the contamination risk?

【Pre-Training Data Engineering 核心考点 5】Synthetic data (Evol-Instruct etc.) and the contamination risk?。技术解析：Definition: Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are ca... 完整推导与工程落地见：https://talentme.airsota.com/resources/tech/roadmap/mindmap/LLM/card/pretrain-data-engineering

预训练数据工程

🎯Core Definition

Pre-training data engineering is the pipeline that sets the model's capability ceiling: cleaning → dedup → quality filtering → domain mix → annealing. Mechanism: MinHash+LSH hashes documents into ~128-bit signatures and splits them into bands; colliding buckets are candidate near-duplicate pairs. Quality filtering scores documents by PPL (perplexity) of a pretrained LM; the domain mix blends Web:Code:Math at a set ratio; the annealing phase (last ~5% of tokens) injects high-quality and synthetic data such as Evol-Instruct's iteratively evolved instructions.

💡Use Cases

data pipelines before pretraining, cleaning RLHF preference data, model reproduction and data-ablation analysis.

⚡Key Problems Solved

trillion-scale corpora without dedup cause repeated memorization, wasted compute and benchmark contamination. MinHash+LSH cuts near-duplicate search from

O(n^2)

to near-linear, scaling to trillions of tokens; PPL filtering removes ~30%-50% of low-quality docs; annealing raises the high-quality share above 50%, gaining several points on downstream benchmarks at the same ~1.4T-token compute budget.