spdl.pipeline.iterate_in_subinterpreter¶

iterate_in_subinterpreter(fn: Callable[[], Iterable[T]], *, buffer_size: int = 3, initializer: Callable[[], None] | Sequence[Callable[[], None]] | None = None, timeout: float | None = None) → Iterable[T][source]¶

[Experimental] Run the given iterable in a subinterpreter.

This function behaves similarly to iterate_in_subprocess(), but uses Python 3.14’s concurrent.interpreters module instead of multiprocessing. Subinterpreters provide isolation while sharing the same process, which can be more lightweight than spawning a separate process.

The subinterpreter is created once and reused across iterations. The returned Iterable supports multiple iterations — each call to iter() (or for ... in) instructs the worker to create a fresh iterator from the underlying iterable without creating a new subinterpreter. Reusing the same worker avoids the overhead of repeated subinterpreter creation and initializer execution.

Note

fn() is called once in the subinterpreter to create the iterable. Each subsequent iter() call creates a fresh iterator by calling iter(iterable) on the same object. If fn() returns a proper Iterable (a class with __iter__ that creates a new iterator each time), re-iteration works as expected.

However, if fn() returns a generator (or any single-use iterator), re-iteration will silently yield no items. This is because a generator is its own iterator — iter(generator) returns self — so once exhausted, calling iter() again returns the same exhausted object. The first iteration will work correctly, but all subsequent iterations will appear empty.

Parameters:

fn – Function that returns an iterator. Use functools.partial() to pass arguments to the function.
buffer_size – Maximum number of items to buffer in the queue.
initializer – Functions executed in the subinterpreter before iteration starts.
timeout – Timeout for inactivity. If the generator function does not yield any item for this amount of time, the subinterpreter is terminated.

Returns:

Iterator over the results of the generator function.

Note

This function requires Python 3.14 or later.
The function and the values yielded by the iterator must be shareable between interpreters.