3.2
3.2 Training Data, Annotation & Synthetic Data
TB/PB-scale pretraining curation, MinHash LSH deduplication, benchmark decontamination, human-in-the-loop annotation, and verifiable synthetic data engines.
PB-Scale Pre-training Curation & Decontamination
MinHash/LSH fuzzy deduplication, FastText quality/language scoring, HTML boilerplate removal, benchmark decontamination (preventing test set leakage), and multimodal pair filtering.
🏢 Companies
Scale AI、Labelbox、Appen、Surge AI、Hugging Face
🛠️ Tech Stack
MinHash LSHDeduplicationDecontaminationFastTextPII RemovalData Quality ScoringFineWeb
💼 Roles & Salary
ML Data Engineer、Data Operations Engineer、Annotation Operations Lead
💰 $180K - $370K / year (ML Data Curation) | ¥480K - ¥1.15M / year
📚 Prerequisites: MinHash & LSH Algorithms • Benchmark Decontamination Techniques • Regex & Text Standardization • Distributed Batch Data Processing
Synthetic Data Engines, Sandbox Verification & Provenance
Teacher-model Evol-Instruct evolutionary scaling, code execution sandbox validation, rejection sampling, human-in-the-loop QA gates, and data provenance/licensing tracking.
🏢 Companies
Scale AI、Gretel.ai、Snorkel AI、Surge AI
🛠️ Tech Stack
Synthetic DataEvol-InstructRejection SamplingExecution FeedbackData ProvenanceHuman-in-the-loopSelf-Instruct
💼 Roles & Salary
Synthetic Data Engineer、ML Data Engineer、Data Quality Engineer
💰 $195K - $410K / year (Synthetic Data R&D) | ¥500K - ¥1.3M / year
📚 Prerequisites: Self-Instruct & Evol-Instruct Methods • Code Sandbox Execution Feedback • Data Diversity & Perplexity Metrics • Data Licensing & Provenance Tracking