spdl.pipeline.run_pipeline_in_subinterpreter¶

run_pipeline_in_subinterpreter(config: PipelineConfig[T], /, *, num_threads: int, max_failures: int | Fraction = -1, report_stats_interval: float = -1, queue_class: type[AsyncQueue] | None = None, task_hook_factory: Callable[[StageInfo], list[TaskHook]] | None = None, background_tasks: list[Callable[[], BackgroundTask]] | None = None, use_thread_output_queue: bool = False, **kwargs: Any) → Iterable[T][source]¶

[Experimental] Run the given Pipeline in a subinterpreter, and iterate on the result.

The returned Iterable supports multiple iterations. The subinterpreter is created once and reused — each call to iter() (or for ... in) builds a fresh Pipeline inside the same subinterpreter without creating a new one. This avoids the overhead of repeated subinterpreter creation and initializer execution on every iteration.

For multi-epoch training, create the iterable once before the epoch loop and iterate it each epoch:

src = run_pipeline_in_subinterpreter(config, num_threads=4)
for epoch in range(num_epochs):
    for batch in src:
        train(batch)

Parameters:

config – The definition of Pipeline.
num_threads – Passed to build_pipeline().
max_failures – Passed to build_pipeline().
report_stats_interval – Passed to build_pipeline().
queue_class – Passed to build_pipeline().
task_hook_factory – Passed to build_pipeline().
background_tasks – Passed to build_pipeline().
kwargs – Passed to iterate_in_subinterpreter().

Yields:

The results yielded from the pipeline.

Note

Random number generators. Only the stdlib random global state is copied from the main interpreter into the subinterpreter; unlike run_pipeline_in_subprocess(), the NumPy and PyTorch global RNG state is not. Two interpreter limitations force this: the restore initializers cross the boundary as call arguments and so must be shareable across interpreters, which the captured NumPy / PyTorch state objects are not; and NumPy and PyTorch cannot be imported in a subinterpreter at all, so a pipeline running here cannot use them regardless. For any NumPy / PyTorch randomness, seed it from within config — e.g. via an SPDL sampler that stores an explicit seed (DistributedRandomSampler) — rather than depending on the main interpreter’s global RNG state.

Changed in version 0.6.0: The subinterpreter now inherits the main interpreter’s stdlib random global state.