spdl.pipeline¶
Overview¶
Implements Pipeline, a generic task execution engine.
API Reference¶
Functions
build_pipeline¶
Build a pipeline from the config. |
is_eof¶
Test whether the input item is EOF sentinel value. |
profile_pipeline¶
[Experimental] Profile pipeline by running pipes separately while changing the concurrency, measuring performance and logging results. |
cache_iterator¶
Caches values from the iterator and returns caches after the given iteration. |
create_task¶
Wrap |
iterate_in_subinterpreter¶
[Experimental] Run the given |
iterate_in_subprocess¶
[Experimental] Run the given |
run_pipeline_in_subprocess¶
Run the given Pipeline in a subprocess, and iterate on the result. |
run_pipeline_in_subinterpreter¶
[Experimental] Run the given Pipeline in a subinterpreter, and iterate on the result. |
Classes
BackgroundTask¶
A background task that runs alongside pipeline stages. |
ProcessGroupResourceUsage¶
Snapshot of resource usage across all processes in the process group. |
ProcessGroupStatsMonitor¶
Background task that spawns a subprocess to collect per-process-group stats. |
ProfileResult¶
A data class contains profiling result, returned by |
ProfileHook¶
A hook object that can be used to execute custom code before and after each stage and pipeline profiling. |
Pipeline¶
Data processing pipeline. |
PipelineBuilder¶
Build |
TaskHook¶
Base class for hooks to be used in the pipeline. |
TaskStatsHook¶
Track the task runtimes and success rate. |
TaskPerfStats¶
Performance statistics of a task measured by |
AsyncQueue¶
Extends |
StageInfo¶
Structured identity for a pipeline stage. |
StatsQueue¶
Measures the time stages are blocked on upstream/downstream stage. |
QueuePerfStats¶
Performance statistics collected by |
Exceptions
PipelineFailure¶
Thrown by |
Others
BackgroundTaskFactory¶
alias of |