spdl.pipeline.PriorityProcessPoolExecutor

class PriorityProcessPoolExecutor(max_workers: int | None = None, **kwargs: Any)

A ProcessPoolExecutor wrapper whose work-ID queue is priority-ordered.

The _work_ids queue determines which pending work items are fed to worker processes first. Replacing it with a priority queue ensures higher-priority items are dispatched first.

Supports the pickle protocol for use with run_pipeline_in_subprocess().

Example:

pool = PriorityProcessPoolExecutor(max_workers=4)

pipeline = (
    PipelineBuilder()
    .add_source(range(100))
    .pipe(cpu_bound_load, executor=pool.get_executor(), concurrency=4)
    .pipe(cpu_bound_decode, executor=pool.get_executor(), concurrency=4)
    .add_sink(3)
    .build(num_threads=1)
)

with pipeline.auto_stop():
    for item in pipeline.get_iterator():
        process(item)

pool.shutdown()

Methods

get_executor(*[, priority])	Create a child executor for a pipeline stage.
shutdown([wait, cancel_futures])

shutdown(wait: bool = True, cancel_futures: bool = False) → None