Thursday 26 November 2020

Use Multiprocessing to update values in a list of class instances

I have seen some suggestions that I should not even do this using multiprocessing, or that's not the way to use multiprocessing (memory sharing), but I'm going to ask anyway and see if I can understand it better.

I have a Person class, and I want to update the age of that person using update_age method. Let's say I have millions of the Person object (instead of 4 shown here), and I want to use Multiprocessing to update the age of all the Person instances I have.

class Person():
    def __init__(self, age):
        self.age = age

    def update_age(self):
        self.age += 10

people = [Person(1), Person(2), Person(3), Person(4)]

p = Pool(4)

results = [p.apply_async(person.update_age, ()) for person in people]

print([res.get() for res in results])

This will give me [None, None, None, None] because the method Person.update_age does not return anything. If I change the class to:

class Person():
    def __init__(self, age):
        self.age = age

    def update_age(self):
        return self.age + 10

I would get the correct response [11, 12, 13, 14]. But that requires me to restructure all the methods in that class. That is a lot of work if I developed the Person class without thinking about scalability. Is there any way for me to keep the original structure of the class, and still be able to use multiprocessing to spread out the workload to all my CPUs?

EDIT: So if I have a method in the parent process like this.

def update_all():
  for person in people:
    person.update_age()

What would be the correct way to update all the "people" using all CPUs?



from Use Multiprocessing to update values in a list of class instances

No comments:

Post a Comment