spdl.pipeline.PriorityThreadPoolExecutor

class PriorityThreadPoolExecutor(max_workers: int | None = None, thread_name_prefix: str = '', initializer: Callable[[...], object] | None = None, initargs: tuple[Any, ...] = ())

A ThreadPoolExecutor wrapper whose internal queue is priority-ordered.

Use get_executor() to create per-stage executors. Executors created later automatically have higher priority (processed first). Priority can be overridden explicitly via get_executor(priority=N).

Supports the pickle protocol for use with run_pipeline_in_subprocess().

Example:

pool = PriorityThreadPoolExecutor(max_workers=4)

pipeline = (
    PipelineBuilder()
    .add_source(range(100))
    .pipe(load, executor=pool.get_executor(), concurrency=4)
    .pipe(decode, executor=pool.get_executor(), concurrency=4)
    .pipe(transform, executor=pool.get_executor(), concurrency=4)
    .add_sink(3)
    .build(num_threads=1)
)

with pipeline.auto_stop():
    for item in pipeline.get_iterator():
        process(item)

pool.shutdown()

Methods

get_executor(*[, priority])	Create a child executor for a pipeline stage.
shutdown([wait, cancel_futures])

shutdown(wait: bool = True, cancel_futures: bool = False) → None