How to do parallel programming in Python?

by

You can use the multiprocessing module. For this case I might use a processing pool:

from multiprocessing import Pool
pool = Pool()
result1 = pool.apply_async(solve1, [A])    # evaluate "solve1(A)" asynchronously
result2 = pool.apply_async(solve2, [B])    # evaluate "solve2(B)" asynchronously
answer1 = result1.get(timeout=10)
answer2 = result2.get(timeout=10)

This will spawn processes that can do generic work for you. Since we did not pass processes, it will spawn one process for each CPU core on your machine. Each CPU core can execute one process simultaneously.

If you want to map a list to a single function you would do this:

args = [A, B]
results = pool.map(solve1, args)

Don’t use threads because the GIL locks any operations on python objects.

More Related Contents:

  1. How do I parallelize a simple Python loop?
  2. Shared-memory objects in multiprocessing
  3. What are the differences between the threading and multiprocessing modules?
  4. Sharing a result queue among several processes
  5. multiprocessing: Understanding logic behind `chunksize`
  6. Make 2 functions run at the same time
  7. multiprocessing.dummy in Python is not utilising 100% cpu
  8. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  9. Same output in different workers in multiprocessing
  10. Parallelize apply after pandas groupby
  11. Using the multiprocessing module for cluster computing
  12. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  13. Parallel Processing in python
  14. Chunking data from a large file for multiprocessing?
  15. Python: Something like `map` that works on threads [closed]
  16. deciding among subprocess, multiprocessing, and thread in Python?
  17. Python multiprocessing.Pool: AttributeError
  18. Modify object in python multiprocessing
  19. Processes stuck in loop with PyInstaller-executable
  20. How to run multiple functions at the same time?
  21. Is there a simple process-based parallel map for python?
  22. OpenMP and Python
  23. multiprocessing pool example does not work and freeze the kernel
  24. python dask DataFrame, support for (trivially parallelizable) row apply?
  25. cython shared memory in cython.parallel.prange – block
  26. Tensorflow and Multiprocessing: Passing Sessions
  27. Proper way to use multiprocessor.Pool in a nested loop
  28. How to parallelize list-comprehension calculations in Python?
  29. aiohttp: rate limiting parallel requests
  30. Python parallel execution with selenium

Leave a Comment