Passing multiple parameters to pool.map() function in Python [duplicate]

by

You can use functools.partial for this (as you suspected):

from functools import partial

def target(lock, iterable_item):
    for item in iterable_item:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    l = multiprocessing.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

Example:

def f(a, b, c):
    print("{} {} {}".format(a, b, c))

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    a = "hi"
    b = "there"
    func = partial(f, a, b)
    pool.map(func, iterable)
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

Output:

hi there 1
hi there 2
hi there 3
hi there 4
hi there 5

More Related Contents:

  1. Can’t pickle when using multiprocessing Pool.map()
  2. Keyboard Interrupts with python’s multiprocessing Pool
  3. Python Process Pool non-daemonic?
  4. Combine Pool.map with shared memory Array in Python multiprocessing
  5. Filling a queue and managing multiprocessing in python
  6. Multiprocessing.Pool makes Numpy matrix multiplication slower
  7. Can I use a multiprocessing Queue in a function called by Pool.imap?
  8. multiprocessing.Pool() slower than just using ordinary functions
  9. How do you pass a Queue reference to a function managed by pool.map_async()?
  10. Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?
  11. How to let Pool.map take a lambda function
  12. Python: Writing to a single file with queue while using multiprocessing Pool
  13. Python multiprocessing – tracking the process of pool.map operation
  14. How to use Python multiprocessing Pool.map to fill numpy array in a for loop
  15. List comprehension vs map
  16. Multiprocessing vs Threading Python [duplicate]
  17. multiprocessing: How do I share a dict among multiple processes?
  18. Python sharing a lock between processes
  19. How to use multiprocessing queue in Python?
  20. Multiprocessing a for loop?
  21. Using Python’s Multiprocessing module to execute simultaneous and separate SEAWAT/MODFLOW model runs
  22. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  23. “chunksize” parameter in multiprocessing.Pool.map
  24. Connection problems with SQLAlchemy and multiple processes
  25. Python multiprocess profiling
  26. What’s the difference between ThreadPool vs Pool in the multiprocessing module?
  27. Mix Python Twisted with multiprocessing?
  28. How to limit number of CPU’s used by a python script w/o terminal or multiprocessing library?
  29. Multiprocessing pool with an iterator
  30. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?

Leave a Comment