Problem
I am writing a piece of software in which I would like to share an object from a certain module. This object should be modifiable from different modules, and within different processes. Consider following (simplified) version of the problem:
Modules
module_shared.py
# Example class with simplified behaviour
class Shared:
def __init__(self):
self.shared = dict()
def set(self, **kwargs):
for key, value in kwargs.items():
self.shared[key] = value
def get(self, *args):
return {key: self.shared[key] for key in args} if args else self.shared
# Module-scope instance of the Shared class
shared = Shared()
module_a.py
from multiprocessing import Process
from time import sleep
import module_shared as ms
def run():
Process(target=run_process).start()
def run_process():
i = 0
while True:
sleep(3)
ms.shared.set(module_a=i)
i+=1
print("Shared from within module_a", ms.shared.get())
module_b.py
from multiprocessing import Process
from time import sleep
import module_shared as ms
def run():
Process(target=run_process).start()
def run_process():
i = 0
while True:
sleep(2)
ms.shared.set(module_b=i)
i-=1
print("Shared from within module_b", ms.shared.get())
module_main.py
import module_a
import module_b
import module_shared as ms
from time import sleep
if __name__ == '__main__':
module_a.run()
module_b.run()
while True:
sleep(5)
print("Shared from within module_main", ms.shared.get())
Output
The output of running module_main is as follows:
Shared from within module_b {'module_b': 0}
Shared from within module_a {'module_a': 0}
Shared from within module_b {'module_b': -1}
Shared from within module_main {}
Shared from within module_a {'module_a': 1}
Shared from within module_b {'module_b': -2}
...
Expected output is as follows:
Shared from within module_b {'module_b': 0}
Shared from within module_a {'module_a': 0, 'module_b': 0}
Shared from within module_b {'module_a': 0, 'module_b': -1}
Shared from within module_main {'module_a': 0, 'module_b': -1}
Shared from within module_a {'module_a': 1, 'module_b': -1}
Shared from within module_b {'module_a': 1, 'module_b': -2}
...
Further explanation
The shared instance is not modified globally because each Process has its own memory space. Initially I have tried fixing it using the Manager from multiprocessing module, however, I have failed to set it up to also be globally accessible (issues with importing module_shared and what is being executed with the import statement). I have also tried using threading, but achieved the same result as with the manager and simply failed to make it globally accessible between the modules.
At the moment the only solution that has worked was passing the shared object as an argument of both run and run_process functions, and only importing module_shared in module_main. I would prefer not to use this idea because when implementing functions in other modules, the programmer would always have to remember to access the shared object through a function parameter (and later call it in main with it) and the code becomes really messy, whereas module imports would make it very clear and easy to use.
I would appreciate any help to achieve my goal. I am happy to use both the multiprocessing and threading libraries, however I would prefer to use the processes. Naturally, in case any simpler (or better) solutions exist, I would be very happy to consider them.
from Globally modifiable object shared across multiple processes and modules
No comments:
Post a Comment