spdl.pipeline.run_pipeline_in_subinterpreter

run_pipeline_in_subinterpreter(config: PipelineConfig[T], /, *, num_threads: int, max_failures: int = -1, report_stats_interval: float = -1, queue_class: type[AsyncQueue] | None = None, task_hook_factory: Callable[[str], list[TaskHook]] | None = None, **kwargs: Any) → Iterable[T]

[Experimental] Run the given Pipeline in a subinterpreter, and iterate on the result.

Parameters:

• config – The definition of Pipeline.

• num_threads – Passed to build_pipeline().

• max_failures – Passed to build_pipeline().

• report_stats_interval – Passed to build_pipeline().

• queue_class – Passed to build_pipeline().

• task_hook_factory – Passed to build_pipeline().

• kwargs – Passed to iterate_in_subinterpreter().

Yields:

The results yielded from the pipeline.

See also

• iterate_in_subinterpreter() implements the logic for manipulating an iterable in a subinterpreter.

• Parallelism and Performance for the context in which this function was created.