spdl.pipeline.iterate_in_subinterpreter

iterate_in_subinterpreter(fn: Callable[[], Iterable[T]], *, buffer_size: int = 3, initializer: Callable[[], None] | Sequence[Callable[[], None]] | None = None, timeout: float | None = None) → Iterable[T][source]

[Experimental] Run the given iterable in a subinterpreter.

This function behaves similarly to iterate_in_subprocess(), but uses Python 3.14’s concurrent.interpreters module instead of multiprocessing. Subinterpreters provide isolation while sharing the same process, which can be more lightweight than spawning a separate process.

Parameters:

  • fn – Function that returns an iterator. Use functools.partial() to pass arguments to the function.

  • buffer_size – Maximum number of items to buffer in the queue.

  • initializer – Functions executed in the subinterpreter before iteration starts.

  • timeout – Timeout for inactivity. If the generator function does not yield any item for this amount of time, the subinterpreter is terminated.

Returns:

Iterator over the results of the generator function.

Note

  • This function requires Python 3.14 or later.

  • The function and the values yielded by the iterator must be shareable between interpreters.

See also

iterate_in_subprocess() for running in a subprocess instead.

Raises:

RuntimeError – If Python version is less than 3.14.