spdl.source.utils.iterate_in_subprocess

iterate_in_subprocess(fn: Callable[[], Iterable[T]], queue_size: int = 64, mp_context: str = 'forkserver', timeout: float | None = None, daemon: bool = False) → Iterator[T]

Run an iterator in a separate process, and yield the results one by one.

Parameters:

• fn – Function that returns an iterator. Use functools.partial() to pass arguments to the function.

• queue_size – Maximum number of items to buffer in the queue.

• mp_context – Context to use for multiprocessing.

• timeout – Timeout for inactivity. If the generator function does not yield any item for this amount of time, the process is terminated.

• daemnon – Whether to run the process as a daemon.

Returns:

Iterator over the results of the generator function.

Note

The function and the values yielded by the iterator of generator must be picklable.