Measuring the performance

Note

Meta employees, please refer to this.

When Pipeline orchestrates the execution of functions, it also keeps track of various timing information.

Such information are exposed as TaskPerfStats and QueuePerfStats.

You can log such information by following steps.

See also

The performance_analysis example demonstrates writing to TensorBoard.

For QueuePerfStats

  1. Subclass StatsQueue and override StatsQueue.interval_stats_callback().†

  2. In the interval_stats_callback method, save the fields of QueuePerfStats to somewhere you can access later. ††

  3. Provide the new class to PipelineBuilder.build() method.

Similarly for TaskPerfStats

  1. Subclass TaskStatsHook and override TaskStatsHook.interval_stats_callback().†

  2. In the interval_stats_callback method, save the fields of TaskPerfStats to somewhere you can access later. ††

  3. Create a factory function that takes a name of the stage functoin and return a list of TaskHook-s applied to the stage.

  4. Provide the factory function to PipelineBuilder.build() method.

Note

  • When overriding the method, make sure that the method does not hold the GIL, otherwise the logging can degrade the pipeline performance.
  • The destination can be anywhere such as remote database, or local file.