I have a generator (or, a list of generators). Let's call them gens
Each generator in gens
is a complicated function that returns the next value of a complicated procedure. Fortunately, they are all independent of one another.
I want to call gen.__next__()
for each element gen
in gens
, and return the resulting values in a list. However, multiprocessing is unhappy with pickling generators.
Is there a fast, simple way to do this in Python? I would like it such that gens
of length m is mapped to n cores locally on my machine, where n could be larger or smaller than m. Each generator should run on a separate core.
If this is possible, can someone provide a minimal example?
from Simple way to parallelize embarrassingly parallelizable generator
No comments:
Post a Comment