Performance Analysis

When optimizing ML model pipeline, it is important to determine if data loading is bottleneck, and where the bottleneck is.

SPDL facilitates the pipeline performance analysis. Pipeline records runtime performance statistics and export them. In this section, we look at how to log the performance statistics and analyze the pipeline performance.

• Overview of pipeline
• Measuring the performance
• Analyzing the performance
  • TaskPerfStats
    • Average Time
    • The number of task executed
  • QueuePerfStats
    • Throughput
    • Data Readiness
    • Queue Get/Put Time
  • Summary