spdl.pipeline.iterate_in_subprocess

iterate_in_subprocess(fn: Callable[[], Iterable[T]], *, buffer_size: int = 3, initializer: Callable[[], None] | None = None, mp_context: str | None = None, timeout: float | None = None, daemon: bool = False) → Iterator[T][source]

Run an iterator in a separate process, and yield the results one by one.

Parameters:

  • fn – Function that returns an iterator. Use functools.partial() to pass arguments to the function.

  • buffer_size – Maximum number of items to buffer in the queue.

  • initializer – A function executed in the subprocess before iteration starts.

  • mp_context – Context to use for multiprocessing. If not specified, a default method is used.

  • timeout – Timeout for inactivity. If the generator function does not yield any item for this amount of time, the process is terminated.

  • daemon – Whether to run the process as a daemon. Use it only for debugging.

Returns:

Iterator over the results of the generator function.

Note

The function and the values yielded by the iterator of generator must be picklable.