Optimization Guide

In this section, we share the methodologies we developed while optimizing production AI training pipelines.

When optimizing a training pipeline, it is important to determine if data loading is a bottleneck, and identify which part of the pipeline is the bottleneck.

One of the goals of SPDL project is to allow users to programmatically determine where the bottleneck is. SPDL achieves this by making the Pipeline observable. You can export the runtime performance statistics of the Pipeline, and analyse them to determine the bottleneck.

We walk through the steps we take to analyse the production system, and examine some cases, and discuss how we approach performance issues.

• Headspace Analysis
  • Estimating the maximum gain with caching
  • Deciding what to work on
• Noisy Neighbour
• Composition of Pipeline
• Analyzing the Performance
  • TaskPerfStats
    • Average Time
    • The number of tasks executed
  • QueuePerfStats
    • Throughput
    • Data Readiness
    • Queue Get/Put Time
  • Summary
• Understanding the performance statistics
  • Single-stage Pipeline
    • Task Execution Time and Throughput
    • Queue Get/Put Time
      • Fast to slow
      • Slow to fast
    • Queue Occupancy (Data Readiness)
  • Performance Characteristics Summary
  • Multi-Stage Pipeline Structure
    • Fast to Slow
    • Slow to fast
    • Introducing the Concurrency
    • Concurrency beyond optimal setting
  • Aggregation
    • Optimal
    • Non-optimal
  • Summary
    • The Effect of Concurrency
    • Special Considerations for Aggregation
    • Real-World Complexity
• Metrics across Ranks
  • Data Readiness
  • Average Download Time
• The Straggler
• Resolving the Bottleneck
  • Adjust the concurrency
  • Restructure the stages
  • Profiling the function
    • Tracer
    • Sampler

Shared-memory arenas

Once measurements identify allocation, inter-process copying or CPU-to-GPU staging as a relevant cost, consider these shared-memory optimizations. Reusable buffers trade some memory-budget and lifetime discipline for less allocation or copying work; measure the application’s results before selecting a backend. Read Shared-memory arenas for the core pool/ring workflow and the later memory-pinning option for GPU transfer. For technical storage-lifetime management, see Why shared arena outputs keep storage alive in Development Notes.

Shared-memory arenas

• Shared-memory arenas