Performance Analysis

When optimizing ML model pipeline, it is important to determine if data loading is bottleneck. SPDL facilitates the pipeline performance analysis.

Pipeline records runtime statistics and report them at the end of the execution. In this section, we look at how to analyze the pipeline performance and optimize the pipeline.

• Is data loading bottleneck?
  • Baseline
  • Bottleneck is in training loop
  • Bottleneck is in data loading
  • Summary
• Which stage is the bottleneck?
  • Task Stats and QPS
  • Tuning Pipeline Performance
  • Summary