spdl.pipeline.defs¶
Overview¶
This module provides building block definitions for Pipeline.
See also
- Example: Pipeline definitions
Illustrates how to build a complex pipeline.
You can build a pipeline by creating a PipelineConfig,
then passing it to spdl.pipeline.build_pipeline() function.
The following step explains the steps to build a pipeline using the building blocks.
Instantiate a
SourceConfigby providing aIterableorAsyncIterable.Using
Pipe(),Aggregate()orDisaggregate(), instantiate a series ofPipeConfig.Instantiate a
SinkConfig.Pass the config objects created in the previous steps to
PipelineConfig.Pass the
PipelineConfigobject tospdl.pipeline.build_pipeline()function.
API Reference¶
Functions
Aggregate¶
Create a |
Disaggregate¶
Create a |
Merge¶
Create a |
PathVariants¶
Create a |
Pipe¶
Create a |
Classes
Aggregator¶
Abstract base class for custom aggregation operations. |
AggregateConfig¶
Configuration for aggregation operation. |
Collate¶
Aggregator that collects items into batches of a specified size. |
DisaggregateConfig¶
Configuration for disaggregation operation. |
ExecutorConfig¶
[Experimental] Base spec for a worker-pool executor. |
InterpreterPoolExecutorConfig¶
[Experimental] A worker-pool executor backed by subinterpreters. |
MergeConfig¶
Merge multiple pipelines into one output queue. |
PathVariantsConfig¶
Configuration for variant path routing. |
PipeConfig¶
A pipe configuration. |
PipelineConfig¶
A pipeline configuration. |
PlacementConfig¶
[Experimental] A region marker designating where the subsequent stages execute. |
ProcessPoolExecutorConfig¶
[Experimental] A worker-pool executor backed by subprocesses. |
SinkConfig¶
A sink configuration. |
SourceConfig¶
A source configuration. |
Others
MAIN_PROCESS¶
Sentinel |