Getting Started

• Building and Running Pipeline
  • Building a Pipeline
  • Running a Pipeline
  • ⚠ Caveats ⚠
• Pipeline Stages
  • Source
  • Processing
  • Sink
  • Advanced: Merging Multiple Pipelines
• Concurrency
  • Stage Concurrency
  • Thread Pool Size
• Pipeline Parallelism
  • Specifying an executor
  • Multi-threading (default)
  • Multi-threading (custom)
  • Multi-processing (stage)
  • Multi-processing (combined)
  • Multi-threading in subprocess
• Collecting Runtime Statistics
• Building High-Performance DataLoaders
  • Understanding the Paradigm Shift
    • PyTorch DataLoader Architecture
    • SPDL Pipeline Architecture
  • The Key Insight: Split I/O and CPU Work
  • Restructuring PyTorch Datasets for SPDL
  • Building a DataLoader-Style Interface
  • Best Practices
  • Comparison with PyTorch DataLoader
  • When to Use SPDL
  • Next Steps