Monday, 18 January 2021

Simple way to parallelize embarrassingly parallelizable generator

I have a generator (or, a list of generators). Let's call them gens

Each generator in gens is a complicated function that returns the next value of a complicated procedure. Fortunately, they are all independent of one another.

I want to call gen.__next__() for each element gen in gens, and return the resulting values in a list. However, multiprocessing is unhappy with pickling generators.

Is there a fast, simple way to do this in Python? I would like it such that gens of length m is mapped to n cores locally on my machine, where n could be larger or smaller than m. Each generator should run on a separate core.

If this is possible, can someone provide a minimal example?



from Simple way to parallelize embarrassingly parallelizable generator

No comments:

Post a Comment