Modify object in python multiprocessing

by

The problem is that when the objects are passed to the worker processes, they are packed up with pickle, shipped to the other process, where they are unpacked and worked on. Your objects aren’t so much passed to the other process, as cloned. You don’t return the objects, so the cloned object are happily modified, and then thrown away.

It looks like this can not be done (Python: Possible to share in-memory data between 2 separate processes) directly.

What you can do is return the modified objects.

import numpy as np
import multiprocessing as mp



class Tester:

    num = 0.0
    name="none"
    def __init__(self,tnum=num, tname=name):
        self.num  = tnum
        self.name = tname

    def __str__(self):
        return '%f %s' % (self.num, self.name)

def mod(test, nn, out_queue):
    print test.num
    test.num = np.random.randn()
    print test.num
    test.name = nn
    out_queue.put(test)




if __name__ == '__main__':       
    num = 10
    out_queue = mp.Queue()
    tests = np.empty(num, dtype=object)
    for it in range(num):
        tests[it] = Tester(tnum=it*1.0)


    print '\n'
    workers = [ mp.Process(target=mod, args=(test, 'some', out_queue) ) for test in tests ]

    for work in workers: work.start()

    for work in workers: work.join()

    res_lst = []
    for j in range(len(workers)):
        res_lst.append(out_queue.get())

    for test in res_lst: print test

This does lead to the interesting observation that because the spawned processes are identical, they all start with the same seed for the random number, so they all generate the same ‘random’ number.

More Related Contents:

  1. Shared-memory objects in multiprocessing
  2. What are the differences between the threading and multiprocessing modules?
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. multiprocessing.dummy in Python is not utilising 100% cpu
  6. Same output in different workers in multiprocessing
  7. Using the multiprocessing module for cluster computing
  8. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  9. Python multiprocessing.Pool: AttributeError
  10. multiprocessing pool example does not work and freeze the kernel
  11. Tensorflow and Multiprocessing: Passing Sessions
  12. How to run functions in parallel?
  13. Dead simple example of using Multiprocessing Queue, Pool and Locking
  14. multiprocessing.Pool: What’s the difference between map_async and imap?
  15. Log output of multiprocessing.Process
  16. pandas multiprocessing apply
  17. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  18. Exception thrown in multiprocessing Pool not detected
  19. How to iterate over consecutive chunks of Pandas dataframe efficiently
  20. multiprocessing.Pool() slower than just using ordinary functions
  21. Python: Something like `map` that works on threads [closed]
  22. deciding among subprocess, multiprocessing, and thread in Python?
  23. Python Multiprocessing Lib Error (AttributeError: __exit__)
  24. Is there a simple process-based parallel map for python?
  25. Show the progress of a Python multiprocessing pool imap_unordered call?
  26. Can I get a return value from multiprocessing.Process?
  27. How to utilize all cores with python multiprocessing
  28. How can I share a class between processes?
  29. Multiprocessing useless with urllib2?
  30. Python multiprocessing linux windows difference

Leave a Comment