Dataset Management & Lineage

Dataset management systematically governs the versioning, lineage, storage, compliance and lifecycle of training data. The core tool is DVC (Data Version Control): data files stay out of Git while the version store keeps only metadata (file hashes, dependencies), and data is stored by content-addressable storage (the hash is the address), giving versioning, branching and rollback equivalent to code. Lineage records the full chain from raw data → cleaned features → training set → model. Compliance and quality cover training-data deduplication (exact hashing or MinHash/LSH approximate), PII redaction, license/copyright review and cross-border rules; by the birthday paradox, the hash-collision probability for exact dedup is $P \approx \frac{n^2}{2^{m+1}}$, where $m$ is the hash width in bits and $n$ the number of samples.

TB-to-PB training sets with millions of files make Git-based management impractical; reproducing experiments demands aligned versions of code, data and model; audits require traceable data provenance; interviews ask about the DVC/Git division of labor, dedup methods and data copyright boundaries.

versus copying and backing up whole datasets, content addressing plus dedup drastically cut storage (identical files stored once, only diffs between versions) and make rollback instant; lineage lets any model output be traced back to its data and code versions for reproducibility and audit; dedup and compliance checks keep duplicated or infringing samples out of training, reducing memorization bias and legal risk.

How does DVC version large datasets? Why keep data files out of Git, and how are metadata and data separated?

Why does data lineage matter? How do you record and trace the data-to-feature-to-model chain?

When to use exact-hash vs MinHash/LSH dedup? What is the hash-collision probability formula?

What does training-data compliance cover? How to implement PII redaction, licensing and cross-border rules?

Data lifecycle: how to design retention, archive and deletion policies, and how do they tie to model versions?