multiprocessing global variable updates not returned to parent

by

When you use multiprocessing to open a second process, an entirely new instance of Python, with its own global state, is created. That global state is not shared, so changes made by child processes to global variables will be invisible to the parent process.

Additionally, most of the abstractions that multiprocessing provides use pickle to transfer data. All data transferred using proxies must be pickleable; that includes all the objects that a Manager provides. Relevant quotations (my emphasis):

Ensure that the arguments to the methods of proxies are picklable.

And (in the Manager section):

Other processes can access the shared objects by using proxies.

Queues also require pickleable data; the docs don’t say so, but a quick test confirms it:

import multiprocessing
import pickle

class Thing(object):
    def __getstate__(self):
        print 'got pickled'
        return self.__dict__
    def __setstate__(self, state):
        print 'got unpickled'
        self.__dict__.update(state)

q = multiprocessing.Queue()
p = multiprocessing.Process(target=q.put, args=(Thing(),))
p.start()
print q.get()
p.join()

Output:

$ python mp.py 
got pickled
got unpickled
<__main__.Thing object at 0x10056b350>

The one approach that might work for you, if you really can’t pickle the data, is to find a way to store it as a ctype object; a reference to the memory can then be passed to a child process. This seems pretty dodgy to me; I’ve never done it. But it might be a possible solution for you.

Given your update, it seems like you need to know a lot more about the internals of a LORR. Is LORR a class? Can you subclass from it? Is it a subclass of something else? What’s its MRO? (Try LORR.__mro__ and post the output if it works.) If it’s a pure python object, it might be possible to subclass it, creating a __setstate__ and a __getstate__ to enable pickling.

Another approach might be to figure out how to get the relevant data out of a LORR instance and pass it via a simple string. Since you say that you really just want to call the methods of the object, why not just do so using Queues to send messages back and forth? In other words, something like this (schematically):

Main Process              Child 1                       Child 2
                          LORR 1                        LORR 2 
child1_in_queue     ->    get message 'foo'
                          call 'foo' method
child1_out_queue    <-    return foo data string
child2_in_queue                   ->                    get message 'bar'
                                                        call 'bar' method
child2_out_queue                  <-                    return bar data string

More Related Contents:

  1. Python multiprocessing PicklingError: Can’t pickle
  2. Use numpy array in shared memory for multiprocessing
  3. Dead simple example of using Multiprocessing Queue, Pool and Locking
  4. Multiprocessing: How to use Pool.map on a function defined in a class?
  5. multiprocessing.Pool: What’s the difference between map_async and imap?
  6. Python Multiprocessing: Handling Child Errors in Parent
  7. multiprocessing in python – sharing large object (e.g. pandas dataframe) between multiple processes
  8. Log output of multiprocessing.Process
  9. pandas multiprocessing apply
  10. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  11. Multiprocessing.Pool makes Numpy matrix multiplication slower
  12. Can I use a multiprocessing Queue in a function called by Pool.imap?
  13. Using the multiprocessing module for cluster computing
  14. multiprocessing.Pool() slower than just using ordinary functions
  15. Accessing an attribute of a multiprocessing Proxy of a class
  16. call multiprocessing in class method Python
  17. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  18. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  19. How to get the return value of a function passed to multiprocessing.Process?
  20. How to share numpy random state of a parent process with child processes?
  21. multiprocessing vs multithreading vs asyncio
  22. How to let Pool.map take a lambda function
  23. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  24. Python multiprocessing – tracking the process of pool.map operation
  25. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  26. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  27. Why multiprocessing.Process behave differently on windows and linux for global object and function arguments
  28. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  29. Multiprocessing pool with an iterator
  30. Using python multiprocessing with different random seed for each process

Leave a Comment