Using the multiprocessing module for cluster computing

by

If by cluster computing you mean distributed memory systems (multiple nodes rather that SMP) then Python’s multiprocessing may not be a suitable choice. It can spawn multiple processes but they will still be bound within a single node.

What you will need is a framework that handles spawing of processes across multiple nodes and provides a mechanism for communication between the processors. (pretty much what MPI does).

See the page on Parallel Processing on the Python wiki for a list of frameworks which will help with cluster computing.

From the list, pp, jug, pyro and celery look like sensible options although I can’t personally vouch for any since I have no experience with any of them (I use mainly MPI).

If ease of installation/use is important, I would start by exploring jug. It’s easy to install, supports common batch cluster systems, and looks well documented.

More Related Contents:

  1. Shared-memory objects in multiprocessing
  2. What are the differences between the threading and multiprocessing modules?
  3. Sharing a result queue among several processes
  4. multiprocessing: Understanding logic behind `chunksize`
  5. multiprocessing.dummy in Python is not utilising 100% cpu
  6. Same output in different workers in multiprocessing
  7. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  8. Python multiprocessing.Pool: AttributeError
  9. Modify object in python multiprocessing
  10. multiprocessing pool example does not work and freeze the kernel
  11. Tensorflow and Multiprocessing: Passing Sessions
  12. How to run functions in parallel?
  13. Dead simple example of using Multiprocessing Queue, Pool and Locking
  14. multiprocessing.Pool: What’s the difference between map_async and imap?
  15. Log output of multiprocessing.Process
  16. pandas multiprocessing apply
  17. Why does Python’s multiprocessing module import __main__ when starting a new process on Windows?
  18. Appending to the same list from different processes using multiprocessing
  19. Exception thrown in multiprocessing Pool not detected
  20. Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?
  21. How do you pass a Queue reference to a function managed by pool.map_async()?
  22. Passing multiple parameters to pool.map() function in Python [duplicate]
  23. Chrome crashes after several hours while multiprocessing using Selenium through Python
  24. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  25. What is the difference between multiprocessing and subprocess?
  26. Python Multiprocessing Lib Error (AttributeError: __exit__)
  27. Is there a simple process-based parallel map for python?
  28. Show the progress of a Python multiprocessing pool imap_unordered call?
  29. Proper way to use multiprocessor.Pool in a nested loop
  30. How to use Python multiprocessing Pool.map to fill numpy array in a for loop

Leave a Comment