Getting Started

• Building and Running Pipeline
  • Building a Pipeline
  • Running a Pipeline
  • ⚠ Caveats ⚠
• Pipeline Stages
  • Source
  • Processing
  • Sink
  • Advanced: Merging Multiple Pipelines
• Concurrency
  • Stage Concurrency
  • Thread Pool Size
• Pipeline Parallelism
  • Specifying an executor
  • Multi-threading (default)
  • Multi-threading (custom)
  • Multi-processing (stage)
    • The cost of crossing a process boundary
  • Multi-processing (region)
  • Running a pipeline in a subprocess
• Execution Models: MT, MTP, and MP
  • Why the choice matters
  • MT – multi-threading
  • MTP – multi-threading in a subprocess
  • MP – multi-processing
  • Choosing between them
• Working Around the GIL
  • Which operations release the GIL?
  • Example: pandas vs polars
  • What if a function does not release the GIL?
  • Which functions hold the GIL?
• Collecting Runtime Statistics
• Building High-Performance DataLoaders
  • Understanding the Paradigm Shift
    • PyTorch DataLoader Architecture
    • SPDL Pipeline Architecture
  • The Key Insight: Split I/O and CPU Work
  • Restructuring PyTorch Datasets for SPDL
  • Building a DataLoader-Style Interface
  • Best Practices
  • Comparison with PyTorch DataLoader
  • When to Use SPDL
  • Going to Production: Isolate the Loader
  • Next Steps