SPDL (Scalable and Performant Data Loading)

Publications

• Featured in PyCoder’s Weekly #706.

• Resolving Data Starvation with Observability in AI Training (PyTorch Conference, 2025-10-22)

• Optimizing Data Loading for Efficient AI Model Training (@Scale: AI & DATA, 2025-06-25)

• Scalable and Performant Data Loading (arXiv, 2025-04-23)

• Introducing SPDL: Faster AI model training with thread-based data loading (Meta Engineering Blog, 2024-11-22)

Citation

Please use the following BibTex for citing our project if you find it useful.

@misc{hira2025scalableperformantdataloading,
   title={Scalable and Performant Data Loading},
   author={Moto Hira and Christian Puhrsch and Valentin Andrei and Roman Malinovskyy and Gael Le Lan and Abhinandan Krishnan and Joseph Cummings and Miguel Martin and Gokul Gunasekaran and Yuta Inoue and Alex J Turner and Raghuraman Krishnamoorthi},
   year={2025},
   eprint={2504.20067},
   archivePrefix={arXiv},
   primaryClass={cs.DC},
   url={https://arxiv.org/abs/2504.20067},
}

Contents

• Overview
  • What is SPDL?
  • What SPDL is NOT
  • How to use SPDL?
• Installation
  • From PyPI
  • From source
  • Requirements
  • Dependencies
  • Optional Dependencies
  • Building with Free-Threaded Python
• Getting Started
  • Building and Running Pipeline
  • Pipeline Stages
  • Concurrency
  • Pipeline Parallelism
  • Collecting Runtime Statistics
  • Building High-Performance DataLoaders
• Introduction to Async I/O
  • Implementing an Orchestration Mechanism
  • The Event Loop
  • The async and await keywords
  • Running Synchronous Functions
• IO Module
  • High-Level Loading Functions
  • Decoding Process Overview
  • Packets and Frames
  • Filter Graphs
  • Advanced Filter Graphs
  • Streaming Decoding
  • Hardware-Accelerated Video Decoding
  • Loading NumPy Arrays
  • Media Encoding
• Optimization Guide
  • Headspace Analysis
  • Noisy Neighbour
  • Composition of Pipeline
  • Analyzing the Performance
  • Understanding the performance statistics
  • Metrics across Ranks
  • The Straggler
  • Resolving the Bottleneck
• Case Studies
  • Parallelism and Performance
  • Data Format and Performance
  • Building Inference Pipeline
• Migration Guide
  • Why migrate to SPDL?
  • Paradigm Shift
  • Migrating from PyTorch DataLoader
• Best Practices
  • Avoid creating intermediate tensors
  • Make Dataset class composable
• Examples
  • Image dataloading
  • Video dataloading
  • Imagenet classification
  • Multi thread preprocessing
  • Streaming nvdec decoding
  • Streaming video processing
  • Performance analysis
  • Performance simulation
  • Hydra integration
  • Pipeline definitions
  • Pipeline profiling
  • Benchmark utils
  • Benchmark numpy
  • Benchmark wav
  • Benchmark tarfile
  • Benchmark video
• Frequently Asked Questions
  • How to work around GIL?
  • What if a function does not release GIL?
  • Which functions hold the GIL?
  • Why Async I/O?

API References

• API Reference
  • spdl.pipeline
  • spdl.pipeline.defs
  • spdl.pipeline.config
  • spdl.dataloader
  • spdl.source
  • spdl.io
  • spdl.io.utils
• API Reference (C++)
  • Class List
  • File List
  • API
• API Index

Development Notes

• Development Notes
  • Implementation detail of Pipeline
  • Remote Iterable Protocol
  • PyTorch CUDA Race Condition in Multi-threading