Data Loading & IO

Training data loading and IO optimization. WebDataset packs samples sequentially into tar shards read as streams, replacing random reads of millions of small files with sequential IO; format choice: TFRecord (fixed-length records plus index, TensorFlow ecosystem), Parquet (columnar, for feature stores and analytics), Arrow (columnar in-memory format, zero-copy across languages). IO is overlapped with compute: the data pipeline (read → decode → augment → batch) runs in parallel with GPU forward/backward, using prefetch to hide IO latency — e.g. `tf.data.Dataset.prefetch(1)` or a multi-worker PyTorch DataLoader with `prefetch_factor`; with ideal overlap the total time is $T_{\text{total}} \approx \max(T_{\text{io}},\, T_{\text{compute}})$ instead of the sum.

with TB-scale datasets the GPU often idles waiting for data (low GPU utilization plus busy CPU/disk points to an IO bottleneck); distributed training needs per-rank shards with global shuffle between epochs; interviews ask how to detect a data bottleneck, when to pick each format, and how prefetch raises throughput.

versus per-file random reads, millions of small files flood inode lookups and disk random IO, cutting throughput by one to two orders of magnitude versus sequential reads; WebDataset's sequential tar-shard streaming suits both local disks and object stores, and multi-worker prefetch plus local caching (NVMe/page cache) hides IO latency behind compute, recovering GPU utilization and shortening training time by the bottleneck ratio.

Test Your Knowledge: Practice Questions for "Data Loading & IO"

Single choice pitfall questions with instant feedback and mistake tracking.