spdl.pipeline¶
Overview¶
Implements Pipeline, a generic task execution engine.
API Reference¶
Functions
cache_iterator¶
Caches values from the iterator and returns caches after the given iteration. |
create_task¶
Wrapper around |
iterate_in_subprocess¶
Run an iterator in a separate process, and yield the results one by one. |
run_pipeline_in_subprocess¶
Run the given Pipeline in a subprocess, and iterate on the result. |
Classes
Pipeline¶
Data processing pipeline. |
PipelineBuilder¶
Build |
TaskHook¶
Base class for hooks to be used in the pipeline. |
TaskStatsHook¶
Track the task runtimes and success rate. |
TaskPerfStats¶
Performance statistics of a task measured by |
AsyncQueue¶
Extends |
StatsQueue¶
Measures the time stages are blocked on upstream/downstream stage. |
QueuePerfStats¶
Performance statistics collected by |
Exceptions
PipelineFailure¶
Thrown by |