How do you pass a Queue reference to a function managed by pool.map_async()?

by

The following code seems to work:

import multiprocessing, time

def task(args):
    count = args[0]
    queue = args[1]
    for i in xrange(count):
        queue.put("%d mississippi" % i)
    return "Done"


def main():
    manager = multiprocessing.Manager()
    q = manager.Queue()
    pool = multiprocessing.Pool()
    result = pool.map_async(task, [(x, q) for x in range(10)])
    time.sleep(1)
    while not q.empty():
        print q.get()
    print result.get()

if __name__ == "__main__":
    main()

Note that the Queue is got from a manager.Queue() rather than multiprocessing.Queue(). Thanks Alex for pointing me in this direction.

More Related Contents:

  1. Filling a queue and managing multiprocessing in python
  2. Can I use a multiprocessing Queue in a function called by Pool.imap?
  3. Python: Writing to a single file with queue while using multiprocessing Pool
  4. Can’t pickle when using multiprocessing Pool.map()
  5. Sharing a result queue among several processes
  6. Keyboard Interrupts with python’s multiprocessing Pool
  7. Python Process Pool non-daemonic?
  8. Combine Pool.map with shared memory Array in Python multiprocessing
  9. Multiprocessing.Pool makes Numpy matrix multiplication slower
  10. Multiprocessing – Pipe vs Queue
  11. multiprocessing.Pool() slower than just using ordinary functions
  12. Passing multiple parameters to pool.map() function in Python [duplicate]
  13. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  14. How to let Pool.map take a lambda function
  15. python multiprocessing: some functions do not return when they are complete (queue material too big)
  16. Python multiprocessing – tracking the process of pool.map operation
  17. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  18. How to use multiprocessing pool.map with multiple arguments
  19. What are the differences between the threading and multiprocessing modules?
  20. Sharing a complex object between processes?
  21. Child processes created with python multiprocessing module won’t print
  22. Simple Python Multiprocessing function in Spyder doesn’t output results
  23. Python multiprocessing example not working
  24. Multiprocessing launching too many instances of Python VM
  25. Modify object in python multiprocessing
  26. Can I get a return value from multiprocessing.Process?
  27. How to utilize all cores with python multiprocessing
  28. How can I share a class between processes?
  29. Multiprocessing useless with urllib2?
  30. Python multiprocessing linux windows difference

Leave a Comment