Wednesday, 15 May 2019

Parallel code not working when function to parallelize is in a different file

I have a simple function that I want to run in parallel. If the function is directly specified in the main function, it all works nicely. But if the very same function is called from a separate Python file (that is created to contains a series of helper functions), the code fails with the error:

A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

I have tried to run this code:

from joblib import Parallel, delayed
import multiprocessing
import otherFile as of

inputs = range(10) 
def processInput(i):
    return i * i

num_cores = multiprocessing.cpu_count()

results1 = Parallel(n_jobs=num_cores)(delayed(processInput)(i) for i in inputs) # this works
results2 = Parallel(n_jobs=num_cores)(delayed(of.processInput)(i) for i in inputs) # this fails

When I call the function processInput() from the of file I have simply copied the same function in that .py file.

def processInput(i):
    return i * i

How can I make the parallelization work if the function I need to call is in a separate .py file?

This is the full error:

results = Parallel(n_jobs=num_cores)(delayed(of.processInput)(i) for i in inputs)
Traceback (most recent call last):

  File "<ipython-input-387-d8dd1dc361a6>", line 1, in <module>
    results = Parallel(n_jobs=num_cores)(delayed(of.processInput)(i) for i in inputs)

  File "C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\joblib\parallel.py", line 934, in __call__
    self.retrieve()

  File "C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))

  File "C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\site-packages\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)

  File "C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py", line 432, in result
    return self.__get_result()

  File "C:\Users\xxxxx\AppData\Local\Continuum\anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.*



from Parallel code not working when function to parallelize is in a different file

1 comment: