Can I use a multiprocessing Queue in a function called by Pool.imap?

by

The trick is to pass the Queue as an argument to the initializer. Appears to work with all the Pool dispatch methods.

import multiprocessing as mp

def f(x):
    f.q.put('Doing: ' + str(x))
    return x*x

def f_init(q):
    f.q = q

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None, f_init, [q])
    results = p.imap(f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

More Related Contents:

  1. Filling a queue and managing multiprocessing in python
  2. How do you pass a Queue reference to a function managed by pool.map_async()?
  3. Python: Writing to a single file with queue while using multiprocessing Pool
  4. Can’t pickle when using multiprocessing Pool.map()
  5. Sharing a result queue among several processes
  6. Keyboard Interrupts with python’s multiprocessing Pool
  7. Python Process Pool non-daemonic?
  8. Combine Pool.map with shared memory Array in Python multiprocessing
  9. Multiprocessing.Pool makes Numpy matrix multiplication slower
  10. Multiprocessing – Pipe vs Queue
  11. multiprocessing.Pool() slower than just using ordinary functions
  12. Passing multiple parameters to pool.map() function in Python [duplicate]
  13. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  14. How to let Pool.map take a lambda function
  15. python multiprocessing: some functions do not return when they are complete (queue material too big)
  16. Python multiprocessing – tracking the process of pool.map operation
  17. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  18. How to run functions in parallel?
  19. Dead simple example of using Multiprocessing Queue, Pool and Locking
  20. multiprocessing.Pool: What’s the difference between map_async and imap?
  21. Solving embarassingly parallel problems using Python multiprocessing
  22. Log output of multiprocessing.Process
  23. pandas multiprocessing apply
  24. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  25. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  26. Chrome crashes after several hours while multiprocessing using Selenium through Python
  27. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  28. What is the difference between multiprocessing and subprocess?
  29. Python Multiprocessing Lib Error (AttributeError: __exit__)
  30. Show the progress of a Python multiprocessing pool imap_unordered call?

Leave a Comment