Pass multiple parameters to concurrent.futures.Executor.map?

by

One argument that is repeated, one argument in c

from itertools import repeat
for result in executor.map(f, repeat(a), c):
    pass

Need to unpack items of c, and can unpack c

from itertools import izip
for result in executor.map(f, *izip(*c)):
    pass

Need to unpack items of c, can’t unpack c

  1. Change f to take a single argument and unpack the argument in the function.

  2. If each item in c has a variable number of members, or you’re calling f only a few times:

    executor.map(lambda args, f=f: f(*args), c)

    It defines a new function that unpacks each item from c and calls f. Using a default argument for f in the lambda makes f local inside the lambda and so reduces lookup time.

  3. If you’ve got a fixed number of arguments, and you need to call f a lot of times:

    from collections import deque
def itemtee(iterable, n=2):
    def gen(it = iter(iterable), items = deque(), next = next):
        popleft = items.popleft
        extend = items.extend
        while True:
            if not items:
                extend(next(it))
            yield popleft()
    return [gen()] * n

executor.map(f, *itemtee(c, n))

Where n is the number of arguments to f. This is adapted from itertools.tee.

More Related Contents:

  1. How can I use threading in Python?
  2. How to build a basic iterator?
  3. How can I iterate over files in a given directory?
  4. Are list-comprehensions and functional functions faster than “for loops”?
  5. Get the first item from an iterable that matches a condition
  6. Iterate over all pairs of consecutive items in a list [duplicate]
  7. Circular list iterator in Python
  8. What’s the shortest way to count the number of items in a generator/iterator?
  9. Solving embarassingly parallel problems using Python multiprocessing
  10. Is there any built-in way to get the length of an iterable in python?
  11. Log output of multiprocessing.Process
  12. Is there an expression for an infinite iterator?
  13. What does “three dots” in Python mean when indexing what looks like a number?
  14. When is not a good time to use python generators?
  15. How to convert an iterable to a stream?
  16. Making a python iterator go backwards?
  17. python groupby behaviour?
  18. split a generator/iterable every n items in python (splitEvery)
  19. Python 3 Map function is not Calling up function
  20. Iterate over the lines of a string
  21. Python: Something like `map` that works on threads [closed]
  22. Performance Advantages to Iterators?
  23. Combine awaitables like Promise.all
  24. How to run multiple functions at the same time?
  25. Why can’t I iterate twice over the same iterator? How can I “reset” the iterator or reuse the data?
  26. How to properly create and run concurrent tasks using python’s asyncio module?
  27. How do I re-map python dict keys
  28. How do I run two python loops concurrently?
  29. How can I count the number of items in an arbitrary iterable (such as a generator)?
  30. Sharing a lock between gunicorn workers

Leave a Comment