spdl.pipeline.run_pipeline_in_subprocess¶
- run_pipeline_in_subprocess(config_or_builder: PipelineConfig[U] | PipelineBuilder[T, U], /, *, num_threads: int, max_failures: int = -1, report_stats_interval: float = -1, queue_class: type[AsyncQueue] | None = None, task_hook_factory: Callable[[str], list[TaskHook]] | None = None, **kwargs: dict[str, Any]) Iterable[T][source]¶
Run the given Pipeline in a subprocess, and iterate on the result.
- Parameters:
config_or_builder – The definition of
Pipeline. Can be either aPipelineConfigorPipelineBuilder.num_threads – Passed to
build_pipeline().max_failures – Passed to
build_pipeline().report_stats_interval – Passed to
build_pipeline().queue_class – Passed to
build_pipeline().task_hook_factory – Passed to
build_pipeline().kwargs – Passed to
iterate_in_subprocess().
- Yields:
The results yielded from the pipeline.
See also
iterate_in_subprocess()implements the logic for manipulating an iterable in a subprocess.Parallelism and Performance for the context in which this function was created.