Python multiprocessing for parallel processes

by

You are correct, they are executing sequentially in your example.

p.join() causes the current thread to block until it is finished executing. You’ll either want to join your processes individually outside of your for loop (e.g., by storing them in a list and then iterating over it) or use something like numpy.Pool and apply_async with a callback. That will also let you add it to your results directly rather than keeping the objects around.

For example:

def f(i):  
    return i*np.identity(4)

if __name__ == '__main__':
    p=Pool(5)
    result = np.zeros((4,4))
    def adder(value):
        global result
        result += value

    for i in range(30):
        p.apply_async(f, args=(i,), callback=adder)
    p.close()
    p.join()
    print result

Closing and then joining the pool at the end ensures that the pool’s processes have completed and the result object is finished being computed. You could also investigate using Pool.imap as a solution to your problem. That particular solution would look something like this:

if __name__ == '__main__':
    p=Pool(5)
    result = np.zeros((4,4))

    im = p.imap_unordered(f, range(30), chunksize=5)

    for x in im:
        result += x

    print result

This is cleaner for your specific situation, but may not be for whatever you are ultimately trying to do.

As to storing all of your varied results, if I understand your question, you can just add it off into a result in the callback method (as above) or item-at-a-time using imap/imap_unordered (which still stores the results, but you’ll clear it as it builds). Then it doesn’t need to be stored for longer than it takes to add to the result.

More Related Contents:

  1. How to run functions in parallel?
  2. multiprocessing.Pool: When to use apply, apply_async or map?
  3. Shared-memory objects in multiprocessing
  4. Shared memory in multiprocessing
  5. Dead simple example of using Multiprocessing Queue, Pool and Locking
  6. Shared variable in python’s multiprocessing
  7. Share Large, Read-Only Numpy Array Between Multiprocessing Processes
  8. multiprocessing.Pool: What’s the difference between map_async and imap?
  9. Python Multiprocessing: Handling Child Errors in Parent
  10. Processing single file from multiple processes
  11. How to use a multiprocessing.Manager()?
  12. Solving embarassingly parallel problems using Python multiprocessing
  13. Log output of multiprocessing.Process
  14. multiprocessing vs multithreading vs asyncio in Python 3
  15. pandas multiprocessing apply
  16. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  17. Multiprocessing : use tqdm to display a progress bar
  18. Exception thrown in multiprocessing Pool not detected
  19. Using the multiprocessing module for cluster computing
  20. multiprocessing.Pool() slower than just using ordinary functions
  21. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  22. How do you pass a Queue reference to a function managed by pool.map_async()?
  23. Chrome crashes after several hours while multiprocessing using Selenium through Python
  24. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  25. How to let Pool.map take a lambda function
  26. Show the progress of a Python multiprocessing pool imap_unordered call?
  27. Python Multiprocessing error: AttributeError: module ‘__main__’ has no attribute ‘__spec__’
  28. Multiprocessing AsyncResult.get() hangs in Python 3.7.2 but not in 3.6
  29. Python multiprocessing application is getting stuck in docker container
  30. How to synchronize a python dict with multiprocessing

Leave a Comment