Is there a simple process-based parallel map for python?

by

I seems like what you need is the map method in multiprocessing.Pool():

map(func, iterable[, chunksize])

A parallel equivalent of the map() built-in function (it supports only
one iterable argument though). It blocks till the result is ready.

This method chops the iterable into a number of chunks which it submits to the 
process pool as separate tasks. The (approximate) size of these chunks can be 
specified by setting chunksize to a positive integ

For example, if you wanted to map this function:

def f(x):
    return x**2

to range(10), you could do it using the built-in map() function:

map(f, range(10))

or using a multiprocessing.Pool() object’s method map():

import multiprocessing
pool = multiprocessing.Pool()
print pool.map(f, range(10))

More Related Contents:

  1. How do I parallelize a simple Python loop?
  2. Shared-memory objects in multiprocessing
  3. What are the differences between the threading and multiprocessing modules?
  4. Sharing a result queue among several processes
  5. multiprocessing: Understanding logic behind `chunksize`
  6. Make 2 functions run at the same time
  7. multiprocessing.dummy in Python is not utilising 100% cpu
  8. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  9. Same output in different workers in multiprocessing
  10. Parallelize apply after pandas groupby
  11. Using the multiprocessing module for cluster computing
  12. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  13. How to iterate over consecutive chunks of Pandas dataframe efficiently
  14. Parallel Processing in python
  15. Chunking data from a large file for multiprocessing?
  16. Python: Something like `map` that works on threads [closed]
  17. deciding among subprocess, multiprocessing, and thread in Python?
  18. Python multiprocessing.Pool: AttributeError
  19. Modify object in python multiprocessing
  20. Processes stuck in loop with PyInstaller-executable
  21. How to run multiple functions at the same time?
  22. OpenMP and Python
  23. multiprocessing pool example does not work and freeze the kernel
  24. cython shared memory in cython.parallel.prange – block
  25. Multiprocessing: use only the physical cores?
  26. Tensorflow and Multiprocessing: Passing Sessions
  27. how to parallelize many (fuzzy) string comparisons using apply in Pandas?
  28. Proper way to use multiprocessor.Pool in a nested loop
  29. How to parallelize list-comprehension calculations in Python?
  30. aiohttp: rate limiting parallel requests

Leave a Comment