SPDL

SPDL (Scalable and Performant Data Loading)

Blog

• Introducing SPDL: Faster AI model training with thread-based data loading (Meta Engineering Blog)

Contents

• Overview
  • What is SPDL?
  • What SPDL is NOT
  • When to use SPDL?
  • Expectations for OSS Support
• Installation
  • From source
  • Requirements
  • Dependencies
  • Optional Dependencies
  • Building with Free-Threaded Python
• Getting Started
  • Building and Running Pipeline
  • Pipeline Stages
  • Concurrency
• Performance Analysis
  • Is data loading bottleneck?
  • Which stage is the bottleneck?
• Usage Guide
  • Why try SPDL?
  • Paradigm Shift
  • Practical Example
• Best Practices
  • Avoid creating intermediate tensors
  • Make Dataset class composable
• Examples
  • image_dataloading
  • video_dataloading
  • imagenet_classification
  • multi_thread_preprocessing
• Frequently Asked Questions
  • How to work around GIL?
  • What if a function does not release GIL?
  • Which functions hold the GIL?
  • Why Async IO?

API References

• API Reference
  • spdl.pipeline
  • spdl.dataloader
  • spdl.source
  • spdl.io
  • spdl.io.utils
• API Reference (C++)
  • Class List
  • File List
  • API
• API Index

Development Notes

• Development Notes
  • Implementation detail of Pipeline