← Back to AI Infrastructure Mind Map
🖥️ AI InfrastructureID: data-loading

Data Loading & IO

训练数据加载与 IO
🎯Core Definition
Training data loading and IO optimization. WebDataset packs samples sequentially into tar shards read as streams, replacing random reads of millions of small files with sequential IO; format choice: TFRecord (fixed-length records plus index, TensorFlow ecosystem), Parquet (columnar, for feature stores and analytics), Arrow (columnar in-memory format, zero-copy across languages). IO is overlapped with compute: the data pipeline (read → decode → augment → batch) runs in parallel with GPU forward/backward, using prefetch to hide IO latency — e.g. `tf.data.Dataset.prefetch(1)` or a multi-worker PyTorch DataLoader with `prefetch_factor`; with ideal overlap the total time is Ttotal≈max⁡(Tio, Tcompute)T_{\text{total}} \approx \max(T_{\text{io}},\, T_{\text{compute}}) instead of the sum.
💡Use Cases
with TB-scale datasets the GPU often idles waiting for data (low GPU utilization plus busy CPU/disk points to an IO bottleneck); distributed training needs per-rank shards with global shuffle between epochs; interviews ask how to detect a data bottleneck, when to pick each format, and how prefetch raises throughput.
⚡Key Problems Solved
versus per-file random reads, millions of small files flood inode lookups and disk random IO, cutting throughput by one to two orders of magnitude versus sequential reads; WebDataset's sequential tar-shard streaming suits both local disks and object stores, and multi-worker prefetch plus local caching (NVMe/page cache) hides IO latency behind compute, recovering GPU utilization and shortening training time by the bottleneck ratio.
🎯Core Interview Exam Points
Fundamentals · Production Scenarios · Deep Dive
Core Def
Explain the core definition, underlying mechanism/math, and key principles of Data Loading & IO.
Use Cases
In production engineering, what are the primary use cases for Data Loading & IO and what core pain points does it solve?
🔥 High-Frequency Deep-Dive Questions
1
Why do many small random reads kill data loading? How does WebDataset's sharded tar fix it?
2
When to use TFRecord vs Parquet vs Arrow?
3
How to tell whether the bottleneck is data loading or compute? Which tools pinpoint it?
4
How do prefetch and multi-worker loading overlap IO with compute? Ideal total time?
5
How are data sharded per rank and globally shuffled across epochs in distributed training?
📖 In-depth Guide:📄 mlops-and-testing →
Updated 2026-08-12
🎯
Verify Mastery: Practice Exam Questions for "Data Loading & IO"
Take pitfall multiple-choice questions with derivation options; mistakes are automatically logged.
🚀 Start Card Practice ➔
← Previous CardModel Weight & Checkpoint StorageNext Card →Data Pipelines & Streaming

🔗 More AI Infrastructure Knowledge Cards

引用出处: TalentMe 450+ 工业级 AI 架构考点知识库 (Tech Vault) · 遵循 CC-BY-4.0 知识共享协议
URL: https://talentme.airsota.com/resources/tech/roadmap/mindmap/AI_Infra/card/data-loading
中文速览与面试要点

训练数据加载与 IO (Data Loading & IO)

定义: 训练数据加载与 IO 优化。WebDataset 把海量样本按序打包成 tar 分片 (shard) 文件流式读取, 用顺序 IO 替代百万级小文件随机读; 格式选择: TFRecord(定长记录 + 索引, TensorFlow 生态)、Parquet(列式, 特征存储与数据分析)、Arrow(列式内存格式, 跨语言零拷贝)。IO 与计算重叠: 数据管线(读取 → 解码 → 增强 → 组 batch)与 GPU 前反向并行, 用预取 (prefetch) 隐藏 IO 延迟, 如 `tf.data.Dataset.prefetch(1)` 或 PyTorch DataLoader 多 worker + `prefetch_factor`; 理想重叠下总耗时 T_ total (T_ io,\, T_ compute), 而不是两者之和。 使用场景: TB 级数据训练时 GPU 常因数据喂不饱而空转(GPU 利用率低 + CPU/磁盘繁忙即可能 IO 瓶颈); 分布式训练需要每个 rank 独立分片、epoch 间全局打乱; 面试常问如何判断数据加载瓶颈、各格式的适用场景、预取如何提升吞吐。 解决的问题: 对比逐文件随机读: 数百万小文件触发海量 inode 查找与磁盘随机 IO, 吞吐比顺序读低一到两个数量级; WebDataset 的 tar 分片顺序流式读取对本地磁盘与对象存储都友好, 配合多 worker 预取与本地缓存(NVMe/页缓存)把 IO 延迟藏在计算背后, GPU 利用率显著回升, 训练时长按瓶颈比例缩短。

核心要点

  • 为什么海量小文件随机读会打爆数据加载? WebDataset 的 shard 机制如何解决?
  • TFRecord / Parquet / Arrow 各自适合什么场景? 如何选择?
  • 如何判断训练瓶颈在数据加载而非算力? 用什么工具定位?
  • 预取与多进程 worker 如何实现 IO 与计算重叠? 理想重叠下的总耗时是多少?
  • 分布式训练中数据如何按 rank 分片、跨 epoch 全局打乱并保证可复现?