Python Multiprocessing with Distributed Cluster

by

If you want a very easy solution, there isn’t one.

However, there is a solution that has the multiprocessing interface — pathos — which has the ability to establish connections to remote servers through a parallel map, and to do multiprocessing.

If you want to have a ssh-tunneled connection, you can do that… or if you are ok with a less secure method, you can do that too.

>>> # establish a ssh tunnel
>>> from pathos.core import connect
>>> tunnel = connect('remote.computer.com', port=1234)
>>> tunnel       
Tunnel('-q -N -L55774:remote.computer.com:1234 remote.computer.com')
>>> tunnel._lport
55774
>>> tunnel._rport
1234
>>> 
>>> # define some function to run in parallel
>>> def sleepy_squared(x):
...   from time import sleep
...   sleep(1.0)
...   return x**2
... 
>>> # build a pool of servers and execute the parallel map
>>> from pathos.pp import ParallelPythonPool as Pool
>>> p = Pool(8, servers=('localhost:55774',))
>>> p.servers
('localhost:55774',)
>>> y = p.map(sleepy_squared, x)
>>> y
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Or, instead you could configure for a direct connection (no ssh)

>>> p = Pool(8, servers=('remote.computer.com:5678',))
# use an asynchronous parallel map
>>> res = p.amap(sleepy_squared, x)
>>> res.get()
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

It’s all a bit finicky, for the remote server to work, you have to start a server running on remote.computer.com at the specified port beforehand — and you have to make sure that both the settings on your localhost and the remote host are going to allow either the direct connection or the ssh-tunneled connection. Plus, you need to have the same version of pathos and of the pathos fork of pp running on each host. Also, for ssh, you need to have ssh-agent running to allow password-less login with ssh.

But then, hopefully it all works… if your function code can be transported over to the remote host with dill.source.importable.

FYI, pathos is long overdue a release, and basically, there are a few bugs and interface changes that need to be resolved before a new stable release is cut.

More Related Contents:

  1. How to use multiprocessing pool.map with multiple arguments
  2. Can’t pickle when using multiprocessing Pool.map()
  3. What are the differences between the threading and multiprocessing modules?
  4. Is shared readonly data copied to different processes for multiprocessing?
  5. Combine Pool.map with shared memory Array in Python multiprocessing
  6. python multiprocessing on windows, if __name__ == “__main__”
  7. Sharing a complex object between processes?
  8. What can multiprocessing and dill do together?
  9. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  10. Filling a queue and managing multiprocessing in python
  11. Child processes created with python multiprocessing module won’t print
  12. Python command line input in a process
  13. What exactly is Python multiprocessing Module’s .join() Method Doing?
  14. yet another confusion with multiprocessing error, ‘module’ object has no attribute ‘f’
  15. Passing multiple parameters to pool.map() function in Python [duplicate]
  16. Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python [duplicate]
  17. Multiprocessing launching too many instances of Python VM
  18. Modify object in python multiprocessing
  19. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  20. How to do multiprocessing in FastAPI
  21. multiprocessing pool example does not work and freeze the kernel
  22. How to process requests from multiiple users using ML model and FastAPI?
  23. Is it possible to multiprocess a function that returns something in Python?
  24. Easiest way to read csv files with multiprocessing in Pandas
  25. Python multiprocessing memory usage
  26. How to signal slots in a GUI from a different process?
  27. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  28. Multiprocessing pool with an iterator
  29. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?
  30. Sharing a lock between gunicorn workers

Leave a Comment