How to parallelize list-comprehension calculations in Python?

by

As Ken said, it can’t, but with 2.6’s multiprocessing module, it’s pretty easy to parallelize computations.

import multiprocessing

try:
    cpus = multiprocessing.cpu_count()
except NotImplementedError:
    cpus = 2   # arbitrary default


def square(n):
    return n * n

pool = multiprocessing.Pool(processes=cpus)
print(pool.map(square, range(1000)))

There are also examples in the documentation that show how to do this using Managers, which should allow for distributed computations as well.

More Related Contents:

  1. how to create a list of lists using list comprehension [closed]
  2. What does “list comprehension” and similar mean? How does it work and how can I use it?
  3. Flattening a shallow list in Python [duplicate]
  4. Is it Pythonic to use list comprehensions for just side effects?
  5. Generator expressions vs. list comprehensions
  6. List comprehension rebinds names even after scope of comprehension. Is this right?
  7. Creating a dictionary from a csv file?
  8. How do I create a list of lambdas (in a list comprehension/for loop)?
  9. Double Iteration in List Comprehension
  10. Sharing a result queue among several processes
  11. Why is a list comprehension so much faster than appending to a list?
  12. How to do parallel programming in Python?
  13. multiprocessing: Understanding logic behind `chunksize`
  14. Make 2 functions run at the same time
  15. List comprehension vs generator expression’s weird timeit results?
  16. multiprocessing.dummy in Python is not utilising 100% cpu
  17. How can I do assignments in a list comprehension?
  18. Comprehension for flattening a sequence of sequences? [duplicate]
  19. How can I use a conditional expression (expression with if and else) in a list comprehension? [duplicate]
  20. Weird behavior: Lambda inside list comprehension [duplicate]
  21. Same output in different workers in multiprocessing
  22. How to return a subset of a list that matches a condition [duplicate]
  23. Why list comprehension is much faster than numpy for multiplying arrays?
  24. How to set local variable in list comprehension?
  25. remove None value from a list without removing the 0 value
  26. How to remove duplicate items from a list using list comprehension? [duplicate]
  27. sum each value in a list of tuples
  28. Proper way to use multiprocessor.Pool in a nested loop
  29. Skip elements on a condition based in a list comprehension in python
  30. Python list comprehension expensive

Leave a Comment