SPDL (Scalable and Performant Data Loading)

Publications

• Introducing SPDL: Faster AI model training with thread-based data loading (Meta Engineering Blog)

• Scalable and Performant Data Loading (arXiv)

Please use the following BibTex for citing our project if you find it useful.

@misc{hira2025scalableperformantdataloading,
   title={Scalable and Performant Data Loading},
   author={Moto Hira and Christian Puhrsch and Valentin Andrei and Roman Malinovskyy and Gael Le Lan and Abhinandan Krishnan and Joseph Cummings and Miguel Martin and Gokul Gunasekaran and Yuta Inoue and Alex J Turner and Raghuraman Krishnamoorthi},
   year={2025},
   eprint={2504.20067},
   archivePrefix={arXiv},
   primaryClass={cs.DC},
   url={https://arxiv.org/abs/2504.20067},
}

Contents

• Overview
  • What is SPDL?
  • What SPDL is NOT
  • How to use SPDL?
• Installation
  • From PyPI
  • From source
  • Requirements
  • Dependencies
  • Optional Dependencies
  • Building with Free-Threaded Python
• Getting Started
  • Building and Running Pipeline
  • Pipeline Stages
  • Concurrency
  • Parallelism
  • Collecting Runtime Statistics
• Optimization Guide
  • Headspace Analysis
  • Noisy Neighbour
  • Building a Pipeline
  • Analyzing the Performance
  • Metrics across Ranks
  • Resolving the Bottleneck
  • Parallelism and Performance
• Migration Guide
  • Why migrate to SPDL?
  • Paradigm Shift
  • Migrating from PyTorch DataLoader
• Best Practices
  • Avoid creating intermediate tensors
  • Make Dataset class composable
• Examples
  • Image dataloading
  • Video dataloading
  • Imagenet classification
  • Multi thread preprocessing
  • Streaming nvdec decoding
  • Streaming video processing
  • Performance analysis
• Frequently Asked Questions
  • How to work around GIL?
  • What if a function does not release GIL?
  • Which functions hold the GIL?
  • Why Async IO?

API References

• API Reference
  • spdl.pipeline
  • spdl.dataloader
  • spdl.source
  • spdl.io
  • spdl.io.utils
• API Reference (C++)
  • Class List
  • File List
  • API
• API Index

Development Notes

• Development Notes
  • Implementation detail of Pipeline