I am trying to map a function call to a pool. I am using the fixes from other similar threads involving calling functions with multiple arguments. I cannot include a more comprehensive example as the train methods for the strategy classes are very long and complicated and the data sets are large:
import multiprocessing as mp
from functools import partial
from numpy import array_split
def call_train(signals, args):
return args[0].train(signals, args[1])
pool = mp.Pool()
chunks = array_split(data.train_signals, pool._processes)
res = pool.map(partial(call_train, [strat, data.train_md]), chunks)
In the above, strat is a python object and both data.train_signals and data.train_md are pandas dataframes.
The error is as follows:
File "/home/jason/PycharmProjects/backtester/core/backtester.py", line 51, in evaluate
res = pool.map(partial(call_train, [strat, data.train_md]), chunks)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
put(task)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle generator objects
from Python Multiprocessing TypeError: can't pickle generator objects
No comments:
Post a Comment