spdl.pipeline.defs.SourceConfig

class SourceConfig(source: Iterable | AsyncIterable, continuous: bool = False)

A source configuration.

A source in Pipeline yields a series of input data, that is going to be processed by downstream pipes.

See also

Example: Pipeline definitions

Illustrates how to build a complex pipeline.

Attributes

continuous	If True, the source continuously re-iterates, injecting a sentinel object representing an epoch boundary between iterations.
source	Generates the series of source data.

continuous: bool = False

If True, the source continuously re-iterates, injecting a sentinel object representing an epoch boundary between iterations. This enables multi-epoch pipeline reuse without rebuilding.

source: Iterable | AsyncIterable

Generates the series of source data.