How can I share a class between processes?

by

multiprocessing.Value isn’t designed to be used with custom classes, it’s supposed to be similar to a multiprocessing.sharedctypes.Value. Instead, you need to create a custom manager and register your class with it. Your life will also be easier if you don’t access value directly, but modify/access it via methods, which will get exported by the default Proxy created for your class by default. Regular attributes (like Counter.value) aren’t, so they aren’t accessible without additional customization. Here’s a working example:

import multiprocessing
from multiprocessing.managers import BaseManager

class MyManager(BaseManager): pass

def Manager():
    m = MyManager()
    m.start()
    return m 

class Counter(object):
  def __init__(self):
    self._value = 0

  def update(self, value):
    self._value += value

  def get_value(self):
      return self._value

MyManager.register('Counter', Counter)

def update(counter_proxy, thread_id):
  counter_proxy.update(1)
  print counter_proxy.get_value(), 't%s' % thread_id, \
    multiprocessing.current_process().name
  return counter_proxy

def main():
  manager = Manager()
  counter = manager.Counter()
  pool = multiprocessing.Pool(multiprocessing.cpu_count())
  for i in range(10):
    pool.apply(func=update, args=(counter, i))
  pool.close()
  pool.join()

  print 'Should be 10 but is %s.' % counter.get_value()

if __name__ == '__main__':
  main()

Output:

1 t0 PoolWorker-2
2 t1 PoolWorker-8
3 t2 PoolWorker-4
4 t3 PoolWorker-5
5 t4 PoolWorker-6
6 t5 PoolWorker-7
7 t6 PoolWorker-3
8 t7 PoolWorker-9
9 t8 PoolWorker-2
10 t9 PoolWorker-8
Should be 10 but is 10.

More Related Contents:

  1. How to use multiprocessing pool.map with multiple arguments
  2. Python multiprocessing PicklingError: Can’t pickle
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. Shared variable in python’s multiprocessing
  6. multiprocessing.Pool: What’s the difference between map_async and imap?
  7. Script using multiprocessing module does not terminate
  8. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  9. Leveraging “Copy-on-Write” to Copy Data to Multiprocessing.Pool() Worker Processes
  10. Python multiprocessing.Pool: AttributeError
  11. How to get the return value of a function passed to multiprocessing.Process?
  12. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  13. how to to terminate process using python’s multiprocessing
  14. multiprocessing pool example does not work and freeze the kernel
  15. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  16. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  17. Creating and updating nested dictionaries and lists inside multiprocessing.Manager object
  18. Python multiprocessing application is getting stuck in docker container
  19. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  20. Understanding Multiprocessing: Shared Memory Management, Locks and Queues in Python
  21. How can I abort a task in a multiprocessing.Pool after a timeout?
  22. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  23. How can I recover the return value of a function passed to multiprocessing.Process?
  24. Why does multiprocessing use only a single core after I import numpy?
  25. How to increment a shared counter from multiple processes?
  26. Same output in different workers in multiprocessing
  27. multiprocessing.Pool – PicklingError: Can’t pickle : attribute lookup thread.lock failed
  28. Globals variables and Python multiprocessing [duplicate]
  29. Python OpenCV multiprocessing cv2.VideoCapture mp4
  30. Python Multiprocessing with Distributed Cluster

Leave a Comment