Data Loading & IO

Training data loading and IO optimization. WebDataset packs samples sequentially into tar shards read as streams, replacing random reads of millions of small files with sequential IO; format choice: TFRecord (fixed-length records plus index, TensorFlow ecosystem), Parquet (columnar, for feature stores and analytics), Arrow (columnar in-memory format, zero-copy across languages). IO is overlapped with compute: the data pipeline (read → decode → augment → batch) runs in parallel with GPU forward/backward, using prefetch to hide IO latency — e.g. `tf.data.Dataset.prefetch(1)` or a multi-worker PyTorch DataLoader with `prefetch_factor`; with ideal overlap the total time is $T_{\text{total}} \approx \max(T_{\text{io}},\, T_{\text{compute}})$ instead of the sum.

with TB-scale datasets the GPU often idles waiting for data (low GPU utilization plus busy CPU/disk points to an IO bottleneck); distributed training needs per-rank shards with global shuffle between epochs; interviews ask how to detect a data bottleneck, when to pick each format, and how prefetch raises throughput.

versus per-file random reads, millions of small files flood inode lookups and disk random IO, cutting throughput by one to two orders of magnitude versus sequential reads; WebDataset's sequential tar-shard streaming suits both local disks and object stores, and multi-worker prefetch plus local caching (NVMe/page cache) hides IO latency behind compute, recovering GPU utilization and shortening training time by the bottleneck ratio.

Why do many small random reads kill data loading? How does WebDataset's sharded tar fix it?

When to use TFRecord vs Parquet vs Arrow?

How to tell whether the bottleneck is data loading or compute? Which tools pinpoint it?

How do prefetch and multi-worker loading overlap IO with compute? Ideal total time?

How are data sharded per rank and globally shuffled across epochs in distributed training?