How to increment a shared counter from multiple processes?

by

The problem is that the counter variable is not shared between your processes: each separate process is creating it’s own local instance and incrementing that.

See this section of the documentation for some techniques you can employ to share state between your processes. In your case you might want to share a Value instance between your workers

Here’s a working version of your example (with some dummy input data). Note it uses global values which I would really try to avoid in practice:

from multiprocessing import Pool, Value
from time import sleep

counter = None

def init(args):
    ''' store the counter for later use '''
    global counter
    counter = args

def analyze_data(args):
    ''' increment the global counter, do something with the input '''
    global counter
    # += operation is not atomic, so we need to get a lock:
    with counter.get_lock():
        counter.value += 1
    print counter.value
    return args * 10

if __name__ == '__main__':
    #inputs = os.listdir(some_directory)

    #
    # initialize a cross-process counter and the input lists
    #
    counter = Value('i', 0)
    inputs = [1, 2, 3, 4]

    #
    # create the pool of workers, ensuring each one receives the counter 
    # as it starts. 
    #
    p = Pool(initializer = init, initargs = (counter, ))
    i = p.map_async(analyze_data, inputs, chunksize = 1)
    i.wait()
    print i.get()

More Related Contents:

  1. Multiprocessing vs Threading Python [duplicate]
  2. multiprocessing: How do I share a dict among multiple processes?
  3. multiprocessing: sharing a large read-only object between processes?
  4. Python sharing a lock between processes
  5. How to use multiprocessing queue in Python?
  6. multiprocessing.dummy in Python is not utilising 100% cpu
  7. Compulsory usage of if __name__==”__main__” in windows while using multiprocessing [duplicate]
  8. Multiprocessing – Pipe vs Queue
  9. Concurrent.futures vs Multiprocessing in Python 3
  10. Parallelizing a Numpy vector operation
  11. multiprocessing.pool.MaybeEncodingError: ‘TypeError(“cannot serialize ‘_io.BufferedReader’ object”,)’
  12. High Memory Usage Using Python Multiprocessing
  13. “chunksize” parameter in multiprocessing.Pool.map
  14. Connection problems with SQLAlchemy and multiple processes
  15. OSError 38 [Errno 38] with multiprocessing
  16. Can multiprocessing Process class be run from IDLE
  17. Python selenium multiprocessing
  18. Python multiprocess profiling
  19. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  20. Globals variables and Python multiprocessing [duplicate]
  21. Using 100% of all cores with the multiprocessing module
  22. Mix Python Twisted with multiprocessing?
  23. Can I get a return value from multiprocessing.Process?
  24. How to utilize all cores with python multiprocessing
  25. Tensorflow and Multiprocessing: Passing Sessions
  26. How to synchronize a python dict with multiprocessing
  27. How can I share a class between processes?
  28. Python multiprocessing for parallel processes
  29. Multiprocessing useless with urllib2?
  30. Python multiprocessing linux windows difference

Leave a Comment