spdl.pipeline¶
Overview¶
Implements Pipeline, a generic task execution engine.
API Reference¶
Functions
create_task¶
Wrapper around |
iterate_in_subprocess¶
Run an iterator in a separate process, and yield the results one by one. |
run_pipeline_in_subprocess¶
Run the given Pipeline in a subprocess, and iterate on the result. |
Classes
Pipeline¶
Data processing pipeline. |
PipelineBuilder¶
Build |
PipelineHook¶
Base class for hooks to be used in the pipeline. |
TaskStatsHook¶
Track the task runtimes and success rate. |
AsyncQueue¶
Extends |
StatsQueue¶
Measures the time stages are blocked on upstream/downstream stage. |
Exceptions
PipelineFailure¶
Thrown by |